Imaging control method and device

ABSTRACT

An imaging control method includes receiving a starting instruction including a flight mode of an unmanned aerial vehicle (“UAV”). The imaging control method also includes controlling the UAV to fly autonomously based on the flight mode. The imaging control method also includes obtaining, in the flight mode, location information of a target object, and obtaining orientation information of the target object relative to the UAV based on the target object recognized from an image captured by an imaging device carried by a gimbal mounted on the UAV. The imaging control method further includes controlling a flight path of the UAV based on the location information and the flight mode, controlling an attitude of the gimbal to render the target object to appear in the image, and controlling the imaging device to record a video in the flight mode, and to transmit video data to a terminal.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of InternationalApplication No. PCT/CN2017/085791, filed on May 24, 2017, the entirecontent of which is incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to the technology field of imageacquisition and, more particularly, to an imaging control method anddevice.

BACKGROUND

As the advancement of unmanned aerial vehicles (“UAV”) aerialphotography technologies, there are more and more types of UAVs on themarket. The UAV aerial photography relates to a series of operations,such as camera settings, gimbal control, joystick control, and imagecomposition and viewfinding. If a user desires to use the UAV to capturesmooth videos with beautiful image composition, the user needs to set aseries of parameters for the camera, gimbal, joystick, and imagecomposition and viewfinding. The process of the control is relativelycomplex. It is difficult for a user who is not familiar with aerialphotography operations to set satisfactory parameters in a short amountof time.

SUMMARY

In accordance with an aspect of the present disclosure, there isprovided an imaging control method. The imaging control method includesreceiving a starting instruction comprising a flight mode of an unmannedaerial vehicle (“UAV”). The imaging control method also includescontrolling the UAV to fly autonomously based on the flight mode. Theimaging control method also includes obtaining, in the flight mode,location information of a target object, and obtaining orientationinformation of the target object relative to the UAV based on the targetobject recognized from an image captured by an imaging device carried bya gimbal mounted on the UAV. The imaging control method also includescontrolling a flight path of the UAV based on the location informationand the flight mode. The imaging control method also includescontrolling an attitude of the gimbal to render the target object toappear in the image captured by the imaging device. The imaging controlmethod further includes controlling the imaging device to record a videoin the flight mode, and to transmit video data to a terminal.

In accordance with another aspect of the present disclosure, there isalso provided an imaging control device. The imaging control deviceincludes a memory configured to store computer-executable instructions.The imaging control device also includes a processor configured toretrieve and execute the computer-executable instructions to perform amethod including receiving a starting instruction comprising a flightmode of an unmanned aerial vehicle (“UAV”). The method also includescontrolling the UAV to fly autonomously based on the flight mode. Themethod also includes obtaining, in the flight mode, location informationof a target object, and obtaining orientation information of the targetobject relative to the UAV based on the target object recognized from animage captured by an imaging device carried by a gimbal mounted on theUAV. The method also includes controlling a flight path of the UAV basedon the location information and the flight mode. The method alsoincludes controlling an attitude of the gimbal to render the targetobject to appear in the image captured by the imaging device. The methodfurther includes controlling the imaging device to record a video in theflight mode, and to transmit video data to a terminal.

In the present disclosure, by obtaining location point informationdetermined based on angle data, a control command carrying imagingparameters may be generated based on the location point information. Thecontrol command may be transmitted to a target device, such that thetarget device may execute an imaging control process based on theimaging parameters, thereby improving the imaging efficiency andflexibility.

BRIEF DESCRIPTION OF THE DRAWINGS

To better describe the technical solutions of the various embodiments ofthe present disclosure, the accompanying drawings showing the variousembodiments will be briefly described. As a person of ordinary skill inthe art would appreciate, the drawings show only some embodiments of thepresent disclosure. Without departing from the scope of the presentdisclosure, those having ordinary skills in the art could derive otherembodiments and drawings based on the disclosed drawings withoutinventive efforts.

FIG. 1 is a flow chart illustrating an imaging control method at theUAV, according to an example embodiment.

FIG. 2a is a schematic illustration of an image coordinate system and afield of view, according to an example embodiment.

FIG. 2b is a schematic illustration of a field of view of an imagingdevice, according to an example embodiment.

FIG. 3 is a schematic illustration of allocation coordinates between theUAV and a target object, according to an example embodiment.

FIG. 4 is a schematic illustration of an image composition, according toan example embodiment.

FIG. 5 is a schematic illustration of an image composition, according toanother example embodiment.

FIG. 6a is a schematic illustration of location relationship between theimaging scene and the imaging device, according to an exampleembodiment.

FIG. 6b is a schematic illustration of location relationship between theimaging scene and the imaging device, according to another exampleembodiment.

FIG. 6c is a schematic illustration of location relationship between theimaging scene and the imaging device, according to another exampleembodiment.

FIG. 7 is schematic diagram of a remote control device, according to anexample embodiment.

FIG. 8 is a flow chart of an imaging control method at the smartterminal side, according to an example embodiment.

FIG. 9 is a flow chart of an imaging control method at the smartterminal side, according to another example embodiment.

FIG. 10 is a flow chart of an imaging control method at the smartterminal side, according to another example embodiment.

FIG. 11 is a schematic diagram of a structure of an imaging controldevice, according to an example embodiment.

FIG. 12 is a schematic diagram of an imaging control device at the UAV,according to an example embodiment.

FIG. 13 is a schematic diagram of an imaging control device at the UAV,according to another example embodiment.

FIG. 14 is a schematic diagram of an imaging control device at the smartterminal side, according to an example embodiment.

FIG. 15 is a schematic diagram of an imaging control device at the smartterminal side, according to another example embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Technical solutions of the present disclosure will be described indetail with reference to the drawings, in which the same numbers referto the same or similar elements unless otherwise specified. It will beappreciated that the described embodiments represent some, rather thanall, of the embodiments of the present disclosure. Other embodimentsconceived or derived by those having ordinary skills in the art based onthe described embodiments without inventive efforts should fall withinthe scope of the present disclosure.

In addition, the singular forms of “a,” “an,” and “the” are intended toinclude the plural forms as well, unless the context indicatesotherwise. The terms “comprise,” “comprising,” “include,” and the likespecify the presence of stated features, steps, operations, elements,and/or components but do not preclude the presence or addition of one ormore other features, steps, operations, elements, components, and/orgroups. The term “and/or” used herein includes any suitable combinationof one or more related items listed. For example, A and/or B can mean Aonly, A and B, and B only. The symbol “/” means “or” between the relateditems separated by the symbol. The phrase “at least one of” A, B, or Cencompasses all combinations of A, B, and C, such as A only, B only, Conly, A and B, B and C, A and C, and A, B, and C. In this regard, Aand/or B can mean at least one of A or B.

Further, when an embodiment illustrated in a drawing shows a singleelement, it is understood that the embodiment may include a plurality ofsuch elements. Likewise, when an embodiment illustrated in a drawingshows a plurality of such elements, it is understood that the embodimentmay include only one such element. The number of elements illustrated inthe drawing is for illustration purposes only, and should not beconstrued as limiting the scope of the embodiment. Moreover, unlessotherwise noted, the embodiments shown in the drawings are not mutuallyexclusive, and they may be combined in any suitable manner. For example,elements shown in one embodiment but not another embodiment maynevertheless be included in the other embodiment.

Next, the imaging control method and device of the present disclosurewill be described in detail with reference to the drawings. When thereis no conflict, the following embodiments and the features in thefollowing embodiments may be combined.

The imaging control method and device may be used for controlling theaerial photography of the UAV or the imaging of other aerial photographydevices, such as an unmanned vehicle having a gimbal, a movable robot,etc.

Using the UAV as an example, the UAV may include a carrier and a load.The carrier may enable the load to rotate around one, two, three or moreaxes. In some embodiments, the carrier may enable the load to movelinearly along one, two, three, or more axes. The axes used for rotationor translation may be perpendicular to one another or may not beperpendicular to one another.

In some embodiments, the load may be rigidly carried by or connected tothe UAV, such that the load may maintain a relative static staterelative to the UAV. For example, the carrier connected to the UAV andthe load may not allow the load to move relative to the UAV. In someembodiments, the load may be directly carried by the UAV and may notneed any carrier.

In some embodiments, the load may include one or multiple sensorsconfigured to monitor or track one or multiple target objects. The loadmay include image capturing devices or imaging devices (e.g., camera,camcorder, infrared imaging device, ultraviolet imaging device, orsimilar device), void capturing devices (e.g., parabolic reflectivemicrophone), infrared imaging device, etc. any suitable sensor may beintegrated to the load to capture visual signals, voice signals,electromagnetic signals, or any other expected signals. The sensor mayprovide a static sensing data (e.g., images) or dynamic sensing data(e.g., videos). The sensor may capture the sensing data in real time orcontinuously capture the sensing data at a high frequency.

In various embodiments, the target object tracked by the UAV may includeany natural or man-made object or texture, such as geological scene(e.g., mountain, plant, valley, lake, river, etc.), buildings,transportation tools (e.g., airplanes, ships, cars, trucks, buses, cargovans, or motorcycles). The target object may include a living body, suchas a human or an animal. The target object may be moving or staticrelative to any suitable reference object. The reference object may be arelatively fixed reference object (e.g., the surrounding environment orthe earth). In some embodiments, the reference object may be a movingreference object (e.g., a moving transportation tool). In variousembodiments, the target object may include a passive target object or anactive target object. The active target object may transmit informationof the target object, such as the GPS location of the target object, tothe UAV. The information may be transmitted in a wireless transmissionmanner from a communication unit of the active target object to acommunication unit of the UAV. The active target object may be anenvironmentally friendly transportation tool, a building, an army, etc.the passive target object may not transmit the information of the targetobject. The passive target object may include a neutral or an enemy'stransportation tool, building, army, etc.

The UAV may be configured to receive control data. The smart terminal 2may be configured to provide the control data. The control data may beconfigured to directly or indirectly control various aspects of the UAV.In some embodiments, the control data may include a flight instructionfor controlling the flight parameters of the UAV. The flight parametersmay include a location, a velocity, a direction, or an attitude of theUAV. The control data may be configured to control the flight of theUAV. In some embodiments, the control data may include instructions forcontrolling a specific part of the UAV. For example, the control datamay include information for controlling operations of a carrier. Forexample, the control data may be configured to control an actuationmechanism of the carrier to cause the load to perform angular movementor linear movement relative to the UAV. In other embodiments, thecontrol data may be configured to control the movement of the carrierthat does not carry a load. In other embodiments, the control data maybe configured to adjust one or more operation parameters of the load,such as capturing a static or dynamic image, varying a focus of thelens, start/shutdown, switching imaging modes, changing imagingresolution, changing focus, changing depth of field, changing exposuretime, changing lens speed, changing viewable angle or field of view,etc. In other embodiments, the control data may be configured to controlthe sensor system (not shown), the communication system (not shown),etc., of the UAV.

In some embodiments, the control data of the smart terminal 2 mayinclude target object information. In some embodiments, the targetobject information may include a specified feature of the target object,such as an initial location (e.g., coordinates) and/or a size of thetarget object in one or more images captured by an imaging devicecarried by the UAV. Additionally or alternatively, the target objectinformation may include information relating to the type of the targetobject, such as the type of the target object or features ofclassification, such as color, texture, style, size, shape, dimension,etc. The target object information may include data representing imagesof the target object, including images of the target object in a fieldof view. The field of view may be defined or formed by the images thatthe imaging device can capture.

The target object information may include expected target objectinformation. The expected target object information may specify anexpected feature that the tracked target object in the images capturedby the imaging device should satisfy. The expected target objectinformation may be configured for adjusting the UAV, the carrier, and/orthe imaging device, such that the tracked target object maintains a sameshape or pattern in one or more images based on the expected targetobject information. For example, the target object may be tracked suchthat the target object in the one or more images captured by the imagingdevice maintains an expected location or size. For example, the expectedlocation of the tracked target object may be close to the center of theimage or may be away from the center. For example, the expected size ofthe tracked target object may include an approximate amount (orquantity) of pixels. The expected target object information and theinitial target object information may be the same or different. Invarious embodiments, the expected target object information may beprovided by the smart terminal 2, or may not be provided by the smartterminal 2. For example, the expected target object information may berecorded in the form of hardcode in a logic circuit that is executed bya processing unit of the UAV, be stored locally in the UAV, and/or bestored in a remote data storage unit, or may be obtained from othersuitable sources.

In some embodiments, at least a portion of the target object information(including specified target object information and the information ofthe type of the target object) may be generated based on use inputthrough the smart terminal 2. Additionally or alternatively, the targetobject information may be generated based on other sources. For example,the information of the type of the target object may include previousimages or data stored in a local or remote data storage unit. The imagesmay be images previously captured by the imaging device carried by theUAV or other devices. The images may be computer generated. Theinformation of the type of the target object may be user selected, ormay be provided by the UAV by default.

The UAV may track one or multiple target objects based on the targetobject information. At least a portion of the tracking or other relateddata processing may be executed by one or multiple processors of theUAV. In some embodiments, the target object information may be used forrecognizing a target object to be tracked by the UAV. The recognition ofthe target object may be executed based on initial target objectinformation. The initial target object information may include aspecified feature of a special target object (e.g., initial coordinatesof the target object in an image captured by the UAV), or a commonfeature of a class of target objects (e.g., the color or texture of thetarget object to be tracked). In some embodiments, the recognition ofthe target object may include comparing two or more images to determine,extract, or match features of the two or more images.

When the target object is recognized, the expected target objectinformation may be used to detect a deviation between the target objectand the expected feature, such as a deviation in the expected locationand/or size. In some embodiments, the current feature or information ofthe target object may be obtained through one or multiple imagescaptured by the UAV. The current target object information may becompared with the expected target object information provided by thesmart terminal 2 to determine a difference. The change in the locationof the target object may be obtained through comparing the coordinatesof the target object in the images (e.g., coordinates of the centerpoint of the target object) and the expected coordinates of the locationof the target object. The change in the size of the target object may beobtained through comparing a size of an area covered by the targetobject (e.g., pixels) with a predetermined size of the target object. Insome embodiments, the change in size may be obtained through detecting adirection, a boundary, or other feature of the target object.

A control signal may be generated based on at least a portion of thedeviation (e.g., through one or multiple processors of the UAV), and anadjustment to roughly correct the deviation may be executed based on thecontrol signal. The adjustment may be used in the images captured by theUAV to roughly maintain one or multiple expected features of the targetobject (e.g., the location or size of the target object). In someembodiments, when the UAV execute a flight instruction (e.g., circlingor moving instructions) provided by a user or a predetermined flightpath, the adjustment may be performed in real time. When the imagingdevice captures one or multiple images, the adjustment may be performedin real time. In some embodiments, the adjustment may be performed basedon other data, such as sensing data obtained by one or multiple sensors(e.g., a proximity sensor or GPS sensor) of the UAV. For example,location information of the tracked target object may be obtainedthrough the proximity sensor, and/or be provided by the target objectitself (e.g., GPS location). In addition to being used for detecting thedeviation, the location information may also be used to execute theadjustment.

The adjustment may relate to the UAV, the carrier, and/or the load(e.g., the imaging device). For example, the adjustment may cause theUAV and/or the load (e.g., imaging device) to change location, attitude,direction, angular velocity or linear velocity, etc. the adjustment maycause the carrier to move the load (e.g., imaging device) around one,two, three or more axes relative to the UAV. Further, the adjustment mayinclude adjusting the zoom, focus, or other operation parameters of theload (e.g., imaging device).

In some embodiments, at least a portion of the adjustment may begenerated based on a type of the deviation. For example, the deviationrelative to the expected location of the target object may entail theUAV and/or the load (e.g., through the carrier) to rotate around one,two, or three rotation axes. As another example, a deviation relative tothe expected size of the target object may entail the UAV to translatealong a suitable axis, and/or to change a focal distance of the imagingdevice (e.g., zoom in or zoom out of the lens). For example, if thecurrent or actual size of the target object is smaller than the expectedsize of the target object, the UAV may need to move closer to the targetobject, and/or the imaging device may need to enlarge the target object.On the other hand, if the current or actual size of the target object isgreater than the expected size of the target object, the UAV may need tomove away from the target object, and/or the imaging device may need toshrink the target object.

In various embodiments, the adjustment that corrects the differencerelative to the expected target object information may be realizedthrough controlling one or multiple controllable objects based on thecontrol signal. The controllable object may be a movable device, acarrier, an imaging device, or any combination thereof. In someembodiments, the controllable object may be selected to execute theadjustment. At least a portion of the control signal may be generatedbased on the configuration and setting of the controllable object. Forexample, if the imaging device is stably carried by the UAV and cannotmove relative to the UAV, then rotating the UAV around two axes canrealize the adjustment including rotation around the two correspondingaxes. At this moment, the imaging device may be directly carried by theUAV, or the imaging device may be carried by the UAV through thecarrier. The carrier may not allow relative movement between the imagingdevice and the UAV. If the carrier allows the imaging device to rotatearound at least one axis relative to the UAV, the adjustment of theabove two axes may be realized through a combination of an adjustment tothe UAV and an adjustment to the carrier. In this situation, the carriermay be controlled to execute rotation around one or both of the two axesthat are to be adjusted. In addition, the UAV may be controlled toexecute the rotation around one or both of the two axes that are to beadjusted. For example, the carrier may be a one-axis gimbal, which mayallow the imaging device to rotate around one of the two axes that areto be adjusted. The UAV may rotate around another one of the two axesthat are to be adjusted. In some embodiments, the carrier may allow theimaging device to rotate around two or more axes relative to the UAV.Then the adjustment of the above two axes may be separately accomplishedthrough the carrier. For example, the carrier may include a two-axis ora three-axis gimbal.

In other embodiments, the adjustment for correcting the size of thetarget object may be realized through controlling the zooming operationof the imaging device (if the imaging device can reach the desiredzooming level), or through controlling the movement of the UAV (to becloser to or away from the target object), or a combination of the twomethods. When executing the adjustment, the processor of the UAV maydetermine which method to select or determine to select the combinationof the two methods. For example, if the imaging device is not configuredto reach a zooming level that is needed to maintain the size of thetarget object in the images, the movement of the UAV may be controlledin place of or in addition to the zooming operation of the imagingdevice.

In some embodiments, the adjustment may consider other restrainingconditions. For example, when the flight path of the UAV ispredetermined, the adjustment may be executed through the carrier and/orthe imaging device, and may not affect the movement of the UAV. Forexample, if a remote terminal is autonomously controlling the flight ofthe UAV through the smart terminal 2, or if the UAV is flying(autonomously or semi-autonomously) along a pre-stored flight path, theflight path of the UAV may be predetermined.

Other restraining conditions may include the maximum and/or minimumpredetermined values, operation parameters, or others of the rotationangle, angular velocity, and/or linear velocity of the UAV, carrier,and/or load (e.g., imaging device). The maximum and/or minimumpredetermined values may be used to display a range of adjustment. Forexample, an angular velocity of the UAV and/or imaging device around aspecified axis may be limited by the maximum angular velocity of theUAV, carrier, and/or load (e.g., imaging device). As another example,the linear velocity of the UAV and/or carrier may be limited by themaximum linear velocity of the UAV, carrier, and/or load (e.g., imagingdevice). As a further example, the adjustment to the focal distance ofthe imaging device may be limited by the maximum and/or minimum focaldistance of the specific imaging device. In some embodiments, the limitsmay be predetermined, or may be based on special configurations of theUAV, carrier, and/or load (e.g., imaging device). In some embodiments,the configurations may be adjustable (e.g., through manufacturer,administrator, or user).

In some embodiments, the UAV may be configured to provide data, and thesmart terminal 2 may be configured to receive data, such as the sensingdata obtained by the sensors of the UAV. The smart terminal 2 may alsobe configured to indicate tracking data or information of a feature ofone or multiple target objects tracked by the UAV. The sensing data mayinclude image data captured by the imaging device carried by the UAV, ordata sensed by other sensors. For example, video streams from the UAVand/or the load (e.g., imaging device) may be transmitted to the smartterminal 2 in real time or in near real time. The sensing data may alsoinclude data obtained by a GPS sensor, a motion sensor, an inertialmeasurement unit (“IMU”) sensor, a proximity sensor, or other sensors.The tracking data may include relative or absolute coordinates or sizeof the target object in the image frames received by the UAV, changes ofthe target object in continuous image frames, GPS coordinates, or otherlocation information of the target object. In some embodiments, thesmart terminal 2 may display the tracked target object using thetracking data (e.g., through a graphical tracking indicator, such as arectangular or square frame surrounding the periphery of the targetobject). In various embodiments, the data received by the smart terminal2 may be unprocessed data (e.g., unprocessed sensing data obtained byvarious sensors) and/or processed data (e.g., tracking data processed bythe one or multiple processors of the UAV).

In some embodiments, the location of the smart terminal 2 may be faraway from the UAV, the carrier, and/or the load. The smart terminal 2may be placed or glued to a supporting platform. Alternatively, thesmart terminal 2 may be a handheld or wearable device. For example, thesmart terminal 2 may include a smart phone, a tablet, a laptop, acomputer, glasses, a glove, a helmet, a microphone, or any suitablecombination thereof.

The smart terminal 2 may be configured to display, through a displaydevice, display data received from the UAV. The display data may includethe sensing data, such as the images obtained by the imaging devicecarried by the UAV. The display data may include the trackinginformation. The tracking information and the image data may beseparately displayed or the tracking information may be superimposed ontop of the image data. For example, the display device may be configuredto display images, with the target object in the images being indicatedby the tracking indicator or displayed with highlight. In someembodiments, the tracking indicator may be a rectangular or squareframe, a circle, or other geometric shape surrounding the tracked targetobject. In some embodiments, when the images and tracking data arereceived from the UAV, and/or when the image data are obtained, theimages and the tracking indicator may be displayed in real time. In someembodiments, the display may be delayed.

The smart terminal 2 may be configured to receive an input of a userthrough an input device. The input device may include a joystick, akeyboard, a mouse, a stylus, a microphone, an image or motion sensor, anIMU, etc. Any suitable user input may interact with the terminal, forexample, manual input of instructions, voice control, gesture control,or location control (e.g., through motion, location, or tilting of theterminal). For example, the smart terminal 2 may be configured to allowthe user to operate the joystick to change a direction or attitude ofthe smart terminal 2, use a keyboard, a mouse, a finger, or a stylus tointeract with a user graphical interface, or use other methods tocontrol the UAV, the carrier, or the load, or any combination thereof.

The smart terminal 2 may be configured to allow a user to input thetarget object information through any suitable method. In someembodiments, the smart terminal 2 may enable a user to directly select atarget object from one or multiple displayed images (e.g., a video or asnapshot). For example, the user may use a finger to directly touch ascreen to select the target object, or use the mouse or the joystick toselect the target object. The user may draw a line to surround thetarget object, touch the target object on the image, or select thetarget object on the image. Computer vision or other techniques may beused to recognize the boundary of the target object. One or multipletarget objects may be selected at one time. In some embodiments, theselected target object may be displayed through a selection indicator toindicate that the user has selected the desired target object to track.In some embodiments, the smart terminal 2 may enable the user to selector input the target object information, such as color, texture, shape,dimension, or other desirable features of the target object. Forexample, the user may input information of the type of the targetobject, select such information through the graphical user interface, oruse other methods. In some embodiments, the target object informationmay be obtained from some data sources rather than from the user. Thedata sources may include a remote or local data storage unit, othercomputing devices connected or communicating with the smart terminal 2,etc.

In some embodiments, the smart terminal 2 may enable the user to selectbetween a manual tracking mode and an automatic tracking mode. When themanual tracking mode is selected, the user may specify a specific targetobject for tracking. For example, the user may manually select thetarget object from the image displayed by the smart terminal 2. Specifictarget object information (e.g., coordinates or size) of the selectedtarget object may be transmitted to the UAV as the initial target objectinformation of the target object. On the other hand, when the automatictracking mode is selected, the user may not specify the specific targetobject to be tracked. The user may, for example, specify descriptiveinformation regarding the type of the target object to be trackedthrough the user interface provided by the smart terminal 2. The UAV mayautomatically recognize the image to be tracked based on the initialtarget object information of the specific target object or thedescriptive information of the type of the target object, and may trackthe recognized image.

In general, providing specific target object information (e.g., theinitial target object information) involves more user control in targetobject tracking and less automatic processing or computing (e.g.,recognition of image or target object). The automatic processing orcomputing may be executed by a processing system provided on the UAV. Onthe other hand, providing the descriptive information of the type of thetarget object involves less user control in the tracking of the targetobject, but more computing executed by the processing system provided onthe UAV. An assignment between the user control and the processingsystem control during a tracking process may be adjusted based onvarious factors, such as the environment of the UAV, the velocity orattitude of the UAV, user preference, a computing capability internal toor external to the UAV (e.g., CPU or memory), etc. for example, when theUAV flies in a relatively complex environment (e.g., there are manybuildings, obstacles, or indoor), more user control may be assigned ascompared to the UAV flying in a relatively simple environment (e.g., anopen space or outdoor). In some embodiments, when the UAV flies at arelatively low sea level location, more user control may be assigned ascompared to flying at a high sea level location. In some embodiments,when the UAV is provided with a high speed processor that can executecomplex computation at a faster speed, more automatic control may beassigned to the UAV. In some embodiments, the assignment between theuser control and the UAV automatic control during the tracking processmay be dynamically adjusted based on the above-described factors.

The control data may be generated based on at least a portion of theuser input. The control data may be generated by the smart terminal 2,the UAV, a third device, or any combination thereof. For example, theuser operation of the joystick or the smart terminal 2, or theinteraction with the graphical user interface may be converted intopredetermined control instructions, to change the state or parameters ofthe UAV, the carrier, or the load. In another embodiment, the userselection of the target object in the image displayed by the terminalmay generate the initial target object information and/or the expectedtarget object information to be tracked, such as the initial and/orexpected location and/or size of the target object. Alternatively oradditionally, the control data may be generated based on informationthat is not user operation, such as a remote or local data storage unit,or another computing device connected with the smart terminal 2, etc.

In some embodiments, the UAV may carry a gimbal. The gimbal may carry animaging device. By controlling a rotation angle of the gimbal around oneor multiple rotation axes, the UAV may be enabled to continuouslycapture images of the target object during a process of the UAV movingtoward a location or in a direction. The image including the targetobject that is captured by the imaging device may be transmitted to aground terminal device through a wireless link. For example, the imageincluding the target object that is captured by the UAV may betransmitted through a wireless link to the smart terminal 2, such as asmart phone, a tablet, etc. These smart terminals 2 may have establishedthe communication link with the UAV or directly with the imaging deviceprior to receiving the image including the target object.

The target object may be an object specified by the user, such as anenvironmental object. The image captured by the imaging device may bedisplayed in a user interface. The user may select an object as thetarget object by clicking operations on the image displayed in the userinterface. For example, the user may select a tree, an animal, or anobject in a region as the target object. In some embodiments, the usermay only input image features of certain objects, such as, inputting afeature of a human face, an appearance feature of a certain type ofobjects. A corresponding processing module 204 may process the images tofind a human or an object corresponding to the image features. The foundhuman or object may be used as the target object for imaging.

In some embodiments, the target object may be a static object, or maynot move within a time period of continuous imaging, or the movingvelocity of the target object in the continuous imaging process is muchsmaller than the moving velocity of the UAV, or the velocity differencebetween the two is smaller than a predetermined value.

In some embodiments, to better realize continuous imaging from multipleangles, the gimbal may be a three-axis gimbal. The gimbal may rotatearound three rotation axes: a yaw axis, a pitch axis, and a roll axis.In some embodiments, the gimbal may be a two-axis gimbal. The gimbal mayrotate around two axes: the pitch axis and the roll axis. The gimbal mayinclude two degrees of freedom: the pitch angle and the roll angle. Tocontrol the attitude of the two-axis gimbal in the yaw direction,attitude change of the two-axis gimbal in the yaw direction may berealized by controlling the yaw direction of the UAV. That is, thegimbal may include another degree of freedom, i.e., the yaw angle of theUAV.

The imaging device may be a device having an imaging acquisitioncapability, such as a camera or an image sensor.

The imaging control method and device of the present disclosure will bedescribed below using the imaging control method and device beingimplemented to control the aerial photography of the UAV as an example.

In one embodiment, the present disclosure provides an imaging controlmethod. The method may be implemented at a UAV 1. In this embodiment, atthe UAV 1, the method may be realized by a dedicated control device, ormay be realized by a flight controller of the UAV, or may be realized bya gimbal controller.

Referring to FIG. 1, the imaging control method may include thefollowing steps:

Step S101: receiving a starting instruction, the starting instructionincluding a flight mode of the UAV.

In this embodiment, the starting instruction may be transmitted to theUAV 1 by the smart terminal 2. The smart terminal 2 may include a userinterface. In some embodiments, the user interface may be provided withan operation button for generating the starting instruction. Theoperation button may be a physical button or a virtual button. In someembodiments, when the user is to control the UAV to autonomously fly,the user may press the operation button. This method provides aconvenient and fast control of the UAV to autonomously fly, and does notrequire the user to operate the joystick to control the flight of theUAV.

In some embodiments, the flight mode may include at least one of thefollowing: a slant line mode, a circling mode, a spiral mode, a roaringmode, or a comet circling mode. Each flight mode may include acorresponding flight strategy (the flight strategy corresponding to eachflight mode will be described in detail in the following step S104). Theflight strategy may be configured to instruct the flight of the UAV,thereby realizing a function of one-button control of the UAV to flybased on the desired flight strategy. This method of controlling theflight of the UAV is more precise, convenient, and does not need to relyon complex joystick controls to realize the flight of the UAV. In someembodiments, the flight mode may also include other flight modes, suchas a straight line flight mode.

In some embodiments, the flight mode may be a default flight mode. Thedefault flight mode may be a predetermined flight mode or a combinationof multiple predetermined flight modes. In some embodiments, after theuser presses the operation button for generating the startinginstruction, the smart terminal 2 may select the default flight mode andgenerate the starting instruction based on the default flight mode.

In some embodiments, the flight mode may be a flight mode input by theuser. In some embodiments, the user may select the flight mode of theUAV based on actual need. In some embodiments, the smart terminal 2 maybe preconfigured with multiple flight modes for the user to select. Theuser may select one or more of the multiple selectable flight modesprovided by the smart terminal 2 based on actual need, therebyinstructing the UAV to realize flights under different flight modes toobtain images from different field of views.

In some embodiments, each flight mode may include at least one of acorresponding flight distance and a flight velocity, thereby instructingthe UAV to automatically accomplish flight under each flight mode basedon the flight distance and/or flight velocity. In some embodiments, theflight distance and the flight velocity corresponding to each flightmode may be determined based on actual needs, thereby satisfying diversedemands from the users.

Step S102: controlling the UAV to fly autonomously based on the flightmode;

In this embodiment, step S102 may be executed after step S101 isexecuted, thereby realizing automatic control of the UAV to accomplish arelatively complex flight path. In some embodiments, the UAV mayautonomously fly based on the flight strategy of the flight mode.

In this embodiment, the method may also include: controlling the imagingdevice to record videos in the flight mode, and to transmit video datato the smart terminal 2, thereby obtaining the video data captured inthe aerial photography by the UAV. In some embodiments, in each flightmode, the UAV may store in real time the current video data captured bythe imaging device (i.e., the raw data stream) and may compress the rawdata stream in real time. The UAV may generate a return video stream andtransmit it to the smart terminal 2, such that the smart terminal 2 maydisplay in real time the images currently captured by the UAV.

Step S103: in the flight mode, obtaining location information of atarget object, and obtaining orientation information of the targetobject relative to the UAV based on the target object recognized from animage captured by the imaging device.

In some embodiments, the location information of the target objectrefers to absolution location information of the target object, forexample, coordinates of the target object in the North East Downcoordinate system. The orientation information of the target objectrelative to the UAV refers to the direction of the target objectrelative to the UAV. In some embodiments, the direction information maynot include distance information between the target object and the UAV.

In this embodiment, combining FIG. 2a , FIG. 4, and FIG. 5, a physicalcoordinate system of an image captured by the imaging device may be setas XOY. The origin of the physical coordinate system may be an opticalaxis location of the imaging device. The physical coordinate system XOYmay include an X axis and a Y axis. The X axis may correspond to the yawdirection of the gimbal. The Y axis may correspond to the pitchdirection of the gimbal.

In some embodiments, obtaining the location information of the targetobject may include the following steps: obtaining an information setincluding at least two groups of imaging information, and determiningthe location information of the target object based on at least twogroups of imaging information selected from the information set. In someembodiments, locations corresponding to imaging location informationincluded in each selected group of imaging information may be different.The imaging information may include imaging location information andimaging angle information when the target object is imaged.

In one embodiment of the present disclosure, after determining thetarget object in the captured image, during the process of UAV movingwhile imaging, analysis and recognition may be performed on the imagebased on image recognition technologies. Specifically, image recognitionmay be performed on each captured image based on features such asgrayscale, texture, etc., to obtain a target object and to continuouslyimage the target object.

During the process of continuously imaging the target object, asituation of losing the target object may occur. Multiple reasons maycontribute to the loss of the target object. Specifically, when thetarget object is blocked by another object, the target object may not befind based on image recognition using features such as the grayscale,texture, etc., which may result in the loss of the target object.Alternatively, if the distance to the target object is relatively faraway after the UAV moves, the features of the target object in thecaptured image such as the grayscale, texture, etc., may be insufficientfor recognizing the target object from the image, which may result inthe loss of the target object. Of course, other situations in which thetarget object is lost may also occur. For example, when the lens of theimaging device is subject to illumination of a strong light, thefeatures such as the grayscale, texture, etc., in the captured image maybecome very weak. In some embodiments, the module configured to performimage recognition processing may malfunction. It should be noted thatthe loss of the target object described above means that the targetobject cannot be determined in the image.

In some embodiments, when detecting that the image including the targetobject satisfies a predetermined condition, imaging information whencapturing the image that satisfies the predetermined condition may berecorded. Specifically, the image including the target object satisfyingthe predetermined condition may include: for an image captured at acertain time, the target object can be accurately recognized from theimage based on image recognition technologies. The imaging informationrecorded when the image is captured may include: the imaging locationinformation and the imaging angle information. The imaging locationinformation may be configured to indicate location information of theimaging device when the imaging device captures the target object. Theimaging location information may be the positioning information of theUAV, such as the GPS coordinates. The imaging angle information of thepresent disclosure may be configured to indicate the orientation of thetarget object relative to the imaging device when the imaging devicecaptures the target object. The orientation may be determined based on acombination of an attitude angle of a gimbal (e.g., the yaw angle and/orthe pitch angle of the gimbal) and the display location of the targetobject in the captured image.

During the movement of the UAV, the present disclosure may detect atleast two images that satisfy the predetermined condition, and mayrecord the corresponding imaging information. The recorded imaginginformation may form an information set, which makes it convenient tocalculate the location information of the target object based o theimaging information, such that when the target object is lost, or whenit is needed to image the object directly based on the location, theuser demand for continuous imaging may be satisfied in a certain degree.In some embodiments, locations corresponding to the imaging locationinformation included in each group of imaging information of theinformation set may be different.

In some embodiments, the imaging location information may includeacquired location coordinates of the UAV. The imaging angle informationmay include an angle calculated based on the attitude information of thegimbal and the location information of the target object in the capturedimage. In some embodiments, with respect to the imaging angleinformation, if the target object is located at a center region of thecaptured image when the target object is captured, then the pitch angleof the imaging angle information may be a pitch angle of the gimbal, andthe yaw angle of the imaging angle information may be the yaw angle ofthe gimbal. Referring to FIG. 2a and FIG. 2b , if the target object isnot at the center region of the captured image, a deviation angle of thetarget object relative to an X axis of the image may be determined basedon a pixel distance dp1 (i.e., d_rows in FIG. 5) of a center point ofthe target object relative to the X axis of the image physicalcoordinate system and a value of a horizontal field of view. Inaddition, a deviation angle of the target object relative to a Y axis ofthe image may be based on a pixel distance dp2 (i.e., d_cols in FIG. 5)of the center point of the target object relative to the Y axis of theimage physical coordinate system and a value of a vertical field ofview. The pitch angle in the imaging angle information may be a sum ofthe pitch angle of the gimbal and the deviation angle relative to the Xaxis of the image. The yaw angle in the imaging angle information may bea sum of the yaw angle of the gimbal and the deviation angle relative tothe Y axis of the image. Specifically, FIG. 2a and FIG. 2b show theimage physical coordinate system, the horizontal field of view (“HFOV”)and the vertical field of view (“VFOV”) of the imaging device. Thedeviation angle relative to the X axis of the image and the deviationangle relative to the Y axis of the image may be obtained based on apixel distance ratio between the pixel distances of the center point ofthe target object relative to the X axis and the Y axis and thecorresponding field of view. In addition, combining FIG. 6a , FIG. 6b ,and FIG. 6c , which shows the location relationship between the imagingdevice and the imaging scene, the relationship between the target objectand the field of view (“FOV”) of the imaging device may be learned by aperson having ordinary skills in the art.

After the information set is obtained, when it is needed to realize thecontrol of the flight path of the UAV based on the location, such aswhen the target object cannot be recognized by image recognition, orwhen a condition for continuously controlling the flight path of the UAVbased on the location is satisfied, at least two groups of imaginginformation may be selected from the information set. Selection rulesfor selecting at least two groups of imaging information from theinformation set may include: selecting imaging information based on aseparation distance calculated based on the imaging location informationin the imaging information; and/or selecting the imaging informationbased on a separation angle calculated based on the imaging angleinformation in the imaging information. Satisfying the condition forcontinuous imaging based on the location may include: receiving acontrol instruction from a user for continuous imaging based on thelocation, or location coordinates of the target object can be relativelyaccurately calculated based on information of the information set thatis already recorded.

The present embodiment uses selecting two groups of imaging informationas an example to describe the calculation of the location information ofthe target object. Specifically, as shown in FIG. 3, in the North EastDown coordinate system, the coordinates of the target object may be t(tx, ty), the imaging location information of the selected first groupof imaging information may be d1 (d1 x, d1 y), the yaw angle in theimaging angle information may be yaw1, the imaging location informationof the selected second group of imaging information may be d2 (d2 x, d2y), the yaw angle in the imaging angle information may be yaw2. Based onthe imaging angle information of the two imaging locations,k1=1/tan(yaw1), k2=1/tan(yaw2) may be calculated to obtain a distance ofd1 to the plane where the target object is located, L1=d1 x−k1*d1 y, anda distance of d2 to the plane where the target object is located, L2=d2x−k2*d2 y. Further calculation may obtain the coordinates of the targetobject t as: tx=k1*ty+L1, ty=(L1−L2)/(k2−k1). In the meantime, the pitchangle of the imaging angle information of the first group of imaginginformation may be pitch1, the pitch angle of the imaging angleinformation of the second group of imaging information may be pitch2.The heights of the target object may be estimated as e1 z, e2 z, wheree1 z=d1 z−L1*tan(pitch1), e2 z=d1 z−L2*tan(pitch2). The height of thetarget object may be calculated based on the estimated heights as tz=(e1z+e2 z)/2. Therefore, the three-dimensional coordinates of the targetobject ultimately obtained may be t (tx, ty, tz).

In the present disclosure, the location information of the target objectmay include the coordinates t calculated above. The d1 and d2 may bepositioning coordinates acquired by the positioning module of the UAV,for example, the GPS coordinates obtained by the GPS positioning moduleof the UAV. The yaw angle and the pitch angle of the imaging angleinformation may be calculated based on the yaw angle of the gimbal andthe distance from the image location of the target object to the Y axisof the image, and the pitch angle of the gimbal and the distance fromthe image location of the target object to the X axis of the image, whenthe image from which the target object can be recognized is captured.The detailed calculation methods can refer to the above correspondingdescriptions of FIG. 2a and FIG. 2 b.

In some embodiments, determining the location information of the targetobject based on the at least two groups of imaging information selectedfrom the information set may include: determining location initialestimation information of at least two target objects based on at leastthree groups of imaging information; determining location information ofeach target object based on the location initial estimation information.Specifically, when determining the location initial estimationinformation based on at least three groups of imaging information, eachlocation initial estimation information may be determined based on anytwo groups of imaging information of the at least three groups ofimaging information. The calculation of the location initial estimationinformation may refer to the calculation methods of the locationinformation of the above embodiments. In this embodiment, the locationinformation of the target object relative to the UAV may be informationrandomly selected from multiple pieces of location initial estimationinformation, or may be an average value of location coordinatescorresponding to the multiple pieces of location initial estimationinformation. The location information may be determined according toother rules. For example, the location information may be locationinitial estimation information calculated based on two groups of imaginginformation that have the largest separation distance and/or the largestseparation angle.

In some embodiments, when a location change amplitude between locationscorresponding to at least two pieces of location initial estimationinformation from the already determined various pieces location initialestimation information satisfies a predetermined change amplituderequirement, then it is determined that a stability condition issatisfied. The location change amplitude may refer to the separationdistance between the locations. Satisfying the location change amplituderequirement may include: multiple separation distances are within apredetermined value range. Based on the location change amplitudebetween two or more pieces of location initial estimation information,it may be determined whether the calculated location estimation of thetarget object is stable. The smaller the location change amplitude, themore accurate the calculated location initial estimation information.Conversely, otherwise, it indicates that under the condition that theselected imaging information is not accurate, there is an inaccurateamount in the location initial estimation information, and the locationinformation cannot be accurately determined. Therefore, the adjustmentof the imaging angle based on the location information cannot beperformed, and the continuous imaging of the target object based on thelocation information cannot be performed.

Further, multiple situations can result in the location change amplitudebetween the multiple pieces of location initial estimation informationto be relatively large. For example, when the target object is in astatic state, and when obtaining the information set, the imaginglocation information or the imaging angle information of one or multiplegroups of imaging information is inaccurate, which results in thecalculated location information to be inaccurate. Therefore, whendetermining the location information of the target object, thecalculation may be performed based on the calculated multiple pieces oflocation initial estimation information. For example, after the multiplepieces of location initial estimation information are averaged, theaverage value may be used as the location information of the targetobject.

The above describes some implementation methods for obtaining thelocation information of the target object. In some example, the locationinformation of the target object obtained may include: positioninginformation of the smart terminal 2. The smart terminal 2 may be aterminal that communicates with the UAV. The location information may bethe positioning information. In some embodiments, the smart terminal 2may be a GPS positioning device carried by the target object. The GPSpositioning device may transmit, at a certain frequency, the detectedpositioning information of the target object to the UAV 1, or may obtainthe positioning information of the target object when the UAV 1 inquiresthe GPS positioning device.

In some embodiments, obtaining the orientation information of the targetobject relative to the UAV based on the target object recognized from animage captured by the imaging device may include: obtaining featureinformation of the target object to be tracked; recognizing the targetobject from the captured image based on the feature information andbased on image recognition technology to obtain the orientationinformation of the target object relative to the UAV.

In some embodiments, the descriptions of recognizing the target objectfrom the captured image based on the image recognition technology mayrefer to the above descriptions, which are not repeated.

Step S104: controlling a flight path of the UAV based on the locationinformation and the flight mode.

In this step, the UAV may fly based on the actual location informationof the target object and based on the flight mode in the startinginstruction, thereby realizing different flight paths to obtain imagesfrom angles that are difficult to photograph, which is more suitable forsatisfying the user demands. This embodiment is particularly suitablefor flight paths that have a stronger regularity. It may be difficult tocontrol the UAV to realize relatively complex and particularly flightpaths that have a strong regularity through manual operations of thejoystick.

In some embodiments, the flight strategy corresponding to the slant linemode may include: based on the location information, controlling the UAVto first fly along a horizontal plane (i.e., in a direction parallel tothe ground) and then to fly along a plane forming a certain angle withthe horizontal plane. The angle may be configured based on actual needs,such as 45°, thereby realizing imaging the target object from differentangles to obtain images with relatively rich content. It should be notedthat controlling the UAV to fly along a horizontal plane means that theUAV only has a horizontal flight velocity, and does not have a verticalflight velocity (i.e., in a direction perpendicular to the ground). Insome embodiments, the step of controlling the UAV to first fly along thehorizontal plane and then to fly along a plane forming a certain anglewith the horizontal plane may include: controlling the UAV to fly alongthe horizontal plane; when it is determined that an angle between a lineconnecting the lowest point of the target object and a center of the UAVand a line connecting the highest point of the target object and thecenter of the UAV is smaller than a predetermined multiple of the fieldof view of the imaging device, then based on the location information,controlling the UAV to fly along the plane having the certain angle withthe horizontal plane. The predetermined multiple may be <1. The aboveprocess may enable the UAV to capture images having a more beautifulimage composition. In some embodiments, controlling the UAV to fly alongthe plane forming the certain angle with the horizontal plane mayinclude: controlling the UAV to fly along a line connecting the targetobject and the UAV in a direction away from the target object. The lineconnecting the target object and the UAV may refer to a line connectingany portion of the target object and any portion of the UAV. In someembodiments, the line connecting the target object and the UAV refers toa line connecting a center location of the target object and a centerlocation of the UAV. A rule for determining the center location of thetarget object and the center location of the UAV may be set based onactual needs. For example, for the center location of the target object,a regular shape (e.g., a rectangle, a square, a pentagon, a circle,etc.) may be used to surround the target object. The center location ofthe regular shape may be the center location of the target object.

In some embodiments, the flight strategy corresponding to the slant linemode may include: controlling the UAV to fly in an S shape curve awayfrom the target object based on the location information to captureimages having more beautiful image compositions. A degree of curvatureof the S shaped curve may be set based on actual needs to satisfy theimaging demands.

In some embodiments, using the ground as a reference, the lowest pointand the highest point of the target object are the location of thetarget object closest to the ground and the location of the targetobject farthest from the ground. The angle between a line connecting thelowest point of the target object and the center of the UAV and a lineconnecting the highest point of the target object and the center of theUAV may be referred to as the angle of the target object relative to theUAV. For example, the target object may be a human being. The angle ofthe human being relative to the UAV is the angle between the lineconnecting the lowest point of the human being and the center of the UAVand the line connecting the highest point of the human being and thecenter of the UAV. In some embodiments, the predetermined multiple maybe ⅓, and the target object may be on the ground. When the angle of thetarget object relative to the UAV is smaller than ⅓ of the field of viewof the imaging device, the UAV may fly along the line connecting thetarget object and the UAV in a direction away from the target object,thereby rendering the horizontal line in the image to appear at the ⅓location of the image (i.e., a pixel distance of the horizontal line tothe top edge of the image occupies ⅓ of the total pixel distance of theimage in the Y direction of the physical coordinate system). In themeantime, the target object also appears in the captured image, therebyobtaining the image having a more beautiful image composition.

In some embodiments, the flight strategy corresponding to the circlingmode may include: based on the location information, controlling the UAVto fly circling around the target object based on a specified distance.In this embodiment, the UAV uses the target object as a center, andcircles the target object to move circumferentially, thereby realizingimaging of the target object from 360° directions. The shape of theflight path for flying circling around the target object may be selectedbased on actual needs. In some embodiments, the flight path for flyingcircling around the target object may be a circle. In some embodiments,the flight path for flying circling around the target object may be anoval. In some embodiments, the flight circling around the target objectmay along a flight path that is similar to a circle or oval. Thespecified distance may indicate a distance of the UAV to the targetobject at each location. In some embodiments, the specified distance maybe a default distance. In some embodiments, the flight strategycorresponding to the circling mode may include a default distance. Insome embodiments, the specified distance may be distance informationinput by the user. That is, the user may set the distance informationfor the UAV to fly circling around the target object based on the actualneeds. Thus, different user demands may be satisfied. In someembodiments, after the user selects the circling mode at the smartterminal 2, the user may input at the smart terminal 2 a specifieddistance corresponding to the circling mode, for indicating the distanceinformation for the UAV to fly circling around the target object. Insome embodiments, a distance between the UAV and the target object at acurrent time instance may be calculated based on the locationinformation of the target object and the positioning information of theUAV at the current time instance, thereby further improving theintelligibility of the UAV.

In some embodiments, the flight strategy corresponding to the spiralmode may include: based on the location information, controlling the UAVto fly circling around the target object in a flight path having a shapeof a Fibonacci spiral, an equal ratio spiral, an equiangular spiral, anArchimedean spiral, or a spiral of other shapes. In this embodiment, theUAV may use the target object as a center to fly based on a flight pathhaving a shape of a Fibonacci spiral, an equal ratio spiral, anequiangular spiral, an Archimedean spiral, or a spiral having othershapes, thereby capturing images having richer content. In someembodiments, in order to image the target object from more angles anddirections, the flight strategy corresponding to the spiral mode mayalso include: based on the location information, while controlling theUAV to fly circling around the target object in a flight path having ashape of a Fibonacci spiral, an equal ratio spiral, an equiangularspiral, an Archimedean spiral, or a spiral of other shapes, in themeantime, also controlling the UAV to ascend or descend perpendicular tothe ground based on a predetermined speed. In this embodiment, bycontrolling the UAV to ascend or descend in the direction perpendicularto the ground, imaging the target object from more angles may berealized, thereby enhancing the richness of the content of the capturedimages. The ascending or descending flight velocity of the UAV may beset based on actual needs. In some embodiments, the UAV may fly, basedon the location information, and based on the flight path having a shapeof a Fibonacci spiral, an equal ratio spiral, an equiangular spiral, oran Archimedean spiral, along a horizontal plane circling around thetarget object. That is, the UAV only has a flight velocity in thehorizontal direction. The flight velocity in the vertical direction maybe zero. Through the disclosed flight, the size of the target object inthe captured image may be changed, thereby enhancing the richness of thecaptured images.

In some embodiments, the flight strategy corresponding to the roaringmode may include: based on the location information, controlling the UAVto fly slantly according to a predetermined angle to a first specifiedlocation relative to the target object, and then controlling the UAV toascend perpendicular to the ground. The predetermined angle, the firstspecified location, and the flight velocity of the UAV ascending may beset based on actual needs, thereby realizing capturing diversifiedimages. In some embodiments, the first specified location refers to alocation having a specified distance to a specified location of thetarget object, and the first specified location may be located in aspecified orientation relative to the specified location of the targetobject. In some embodiments, the first specified location may be set bythe user based on the actual needs. In some embodiments, controlling theUAV to fly slantly according to a predetermined angle to the firstpredetermined location of the target object may include: controlling theUAV to fly in a direction moving closer to the target object to thefirst specified location. In some embodiments, controlling the UAV tofly slantly according to the predetermined angle to the first specifiedlocation of the target object may include: controlling the UAV to fly tothe first specified location in a direction moving away from the targetobject.

In addition, in the roaring mode, the UAV may be controlled to fly fromany starting point (i.e., the current location of the UAV) to the firstspecified location. Alternatively, the UAV may be controlled to fly to aspecified starting point, and then be controlled to fly from thespecified starting point to the first specified location. It should benoted that in the situation where the UAV is first controlled to fly tothe specified starting point and then controlled to fly from thespecified starting point to the first specified location, the imagingdevice of the UAV starts video recording after the UAV arrives at thespecified starting point.

In some embodiments, the flight strategy corresponding to the cometcircling mode may include: based on the location information,controlling the UAV to fly to a second specified location in a directionmoving closer to the target object, and after circling around the targetobject starting from the second specified location, fly away from thetarget object. In some embodiments, the second specified location may beset based on actual needs. For example, the second specified locationmay be a location having a specified distance to a specified location ofthe target object, and the second specified location may be located in aspecified orientation of the specified location of the target object. Asa result, diversified images can be captured. In addition, in thisembodiment, the number of circles that the UAV circles around the targetobject after flying to the second specified location may be set based onactual needs, such as one circle, multiple circles, or less than onecircle.

In some embodiments, in the comet circling mode, the UAV may becontrolled to fly from any starting point (i.e., the current location ofthe UAV) in a direction moving closer to the target object to arrive atthe second specified location, and to fly away from the target objectafter circling around the target object from the second specifiedlocation.

In some embodiments, in the comet circling mode, the UAV may becontrolled to fly to a specified starting point, and then be controlledto fly from the specified starting point in a direction moving closer tothe target object until it arrives at the second specified location. TheUAV may be further controlled to fly away from the target object afterflying circling around the target object from the second specifiedlocation. In this embodiment, the imaging device of the UAV may startvideo recording after the UAV arrives at the specified starting point.In addition, in this embodiment, the flight path of the UAV may use thetarget object as a base point, or may be controlled based on coordinatesin the global coordinate system.

In the above embodiment, controlling the UAV to implement the flightpath corresponding to the flight strategy is based on the premise thatthe target object is in the image. In other embodiments, after thelocation information of the target object is obtained, the aircraft maypre-fly along a segment of a flight path without facing the targetobject, and then fly facing the location indicated by the locationinformation of the target object, thereby satisfying different flightdemands.

Step S105: based on the orientation information, controlling attitude ofa gimbal to render the target object to appear in an image captured byan imaging device.

In some embodiments, the attitude of the gimbal may be controlled suchthat the target object consistently appears at a predetermined locationin the captured images, thereby rendering the target object toconsistently appear in the captured images. The orientation informationmay be the predetermined location. In some embodiments, the targetobject being located in the predetermined location of the capturedimages means that the specified location of the target object is at thepredetermined location in the captured images. In some embodiments, thespecified location of the target object is the center location of thetarget object. The predetermined location where the target object islocated in the captured images is the predetermined location in thecaptured images where the center location of the target object islocated. In some embodiments, the predetermined location may begenerated by the user directly clicking any location in a region of auser interface of the smart terminal 2 that is configured for displayingthe captured images. That is, the specified location is a clickinglocation input by the user at the user interface. In some embodiments,the predetermined location may be selected as a default location thespecified location of the target object is displayed in the capturedimages.

In some embodiments, to obtain better image composition effect, theattitude of the gimbal may be controlled such that the size of thetarget object in the captured images maintains a predetermined size. Theorientation information may be the center location information of thepredetermined size or other location information (e.g., vertex angleinformation) in a region corresponding to the predetermined size. Inthis embodiment, the size of the target object in the captured images isa production of a pixel height and a pixel width of the target object inthe captured images. In some embodiments, the predetermined size may bea size frame directly input by the user at the user interface of thesmart terminal 2. In some embodiments, the predetermined size may be adefault size frame. Regardless of whether the predetermined size is asize frame set by the user or a default size frame, in subsequentimaging processes, the target object is located in the size frame in thecaptured images. In some embodiments, the size of the size frame may beconfigured to be just enough to surround the target object in thecaptured images, thereby satisfying an image composition demand of theuser. In some embodiments, the size frame may have a shape of arectangle, a square, etc.

In some embodiments, after the flight path of the UAV is determined, theattitude of the gimbal may be controlled based on a deviation of theactual location of the target object in the images relative to theexpected display location, such that the target object is displayed atthe expected display location (i.e., the predetermined location) in thecaptured images. That is, the location of the target object displayed inthe captured images may be maintained at the predetermined location.Specifically, if there is a left-right deviation in the actual locationof the target object in the captured images relative to the expecteddisplay location, the yaw angle of the gimbal may be controlled tomaintain the target object at the expected display location; if there isan up-down deviation in the actual location of the target object in thecaptured images relative to the expected display location, the pitchangle of the gimbal may be controlled to maintain the target object atthe expected display location.

In some embodiments, location coordinates where the center location ofthe target object to be tracked may be expected to be displayed may be P(u, v), where u is the pixel coordinates of the X axis, and v is thepixel coordinates of the Y axis. The size of the image may be (W, H),where W is the pixel width of the image, H is the pixel height of theimage. If the upper left corner of the image is set as the origin, thenthe angular velocity Yx of the gimbal rotating around the yaw axis maybe:

Yx=*(u−w/2),

where μ is a constant, and μ∈R (R represents real numbers);

the angular velocity Yy of the gimbal rotating around the pitch axis maybe:

Yyω*(v−h/2),

where ω is a constant, and ω∈R.

To maintain the size of the target object in the captured images to bethe predetermined size, after the flight path of the UAV is determined,the focal length of the imaging device may be adjusted based on the sizeof the target object in the captured images. In some embodiments, at theinitial time instance (e.g., a time instance before the smart terminal 2transmits a starting instruction to the UAV 1), the pixel area (i.e.,the predetermined size) of the target object in the captured images is S(S is defined as the pixel height of the target object times the pixelwidth of the target object). During the tracking of the target object,the pixel area of the target object in the captured images may be s.Then the adjustment speed F of the focal length of the imaging devicemay be:

F=γ*(1−s/S),

where γ is a constant, and γ∈R (R represents real numbers).

In some embodiments, during the tracking of the target object, if thepixel area of the target object in the captured images is smaller than apredetermined pixel area of a predetermined size, then the focal lengthof the imaging device may be adjusted to be longer; otherwise, the focallength of the imaging device may be adjusted to be shorter.

In some embodiments, controlling the attitude of the gimbal may include:controlling at least one of the pitch angle, the yaw angle, or the rollangle, and controlling the location of the target object in the capturedimages. In some embodiments, the gimbal may be a three-axis gimbal. Theattitude of the gimbal may be changed by controlling at least one of thepitch axis, yaw axis, and roll axis of the three-axis gimbal.

In some embodiments, the gimbal and the UAV may be fixed relative to oneanother in a heading (i.e., flight direction) axis. Controlling theattitude of the gimbal may include: controlling the pitch angle and/orthe roll angle of the gimbal; controlling a heading angle of the UAV tocontrol the yaw angle of the gimbal. In some embodiments, the gimbal mayhave two degrees of freedom: the pitch angle and the roll angle. Anotherdegree of freedom, i.e., the yaw angle, of the gimbal may be replaced bythe heading angle of the UAV. Thus, the control of the yaw angle of thegimbal may be realized by controlling the heading angle of the UAV.

In some embodiments, to realize the aesthetics of the image compositionof the captured images, the process of determining the pitch angleand/or the yaw angle of the gimbal may include: determining at least oneof the pitch angle or the yaw angle of the gimbal based on an expecteddisplay location of a specified location of a background identificationin the captured images. In this embodiment, by setting the expecteddisplay location of the specified location of the backgroundidentification in the captured images, diversified image compositiondemands may be satisfied, thereby enhancing the richness and theaesthetics of the captured images. The background identification mayinclude at least one of a ground, a sky, a sea surface, a building, orother background identification. In FIG. 4, using the ground as anexample of the background identification, the user may set thehorizontal line to be substantially perpendicular to the Y axis of thephysical coordinate system of the captured image and may be located atabout ⅓ of the Y axis. The UAV may calculate the pitch angle of thegimbal based on the expected display location of the horizontal line inthe captured images, thereby controlling the pitch angle of the gimbal,such that the horizontal line is displayed at the ⅓ location of the Yaxis in the captured images, to obtain better image compositions.

In some feasible embodiments, determining at least one of the pitchangle or the yaw angle of the gimbal based on the expected displaylocation of the specified location of the background identification inthe captured images may include: obtaining a first total pixel distanceof a captured image in a first direction and a pixel distance of theexpected display location of the specified location of the backgroundidentification in the captured image to an edge of the image in thefirst direction, where the first direction may correspond to the pitchdirection or the yaw direction of the gimbal; determining the pitchangle and/or the yaw angle of the gimbal based on the first total pixeldistance, the pixel distance, and the size of the vertical field of viewor the horizontal field of view of the imaging device. In someembodiments, the pixel distance of the upper edge in the firstdirection, i.e., the Y axis direction of the captured image, may be row,the first total pixel distance (i.e., the image height) may be row_size,the vertical field of view of the camera may be VFOV, then, undersimplified conditions, the pitch angle of the gimbal may be calculatedas:

pitch=(row_size/row_size−0.5)*VFOV.

In some embodiments, the location of the background identification inthe captured images may not be set. For example, if the imagecomposition is a high angle shot without a horizontal line, the processfor determining the pitch angle and/or the yaw angle of the gimbal mayinclude: obtaining an elevation angle and/or a horizontal angle of apredetermined imaging location; determining a deviation angle of thetarget object relative to a center line (i.e., the X axis or the Y axisof the physical coordinate system in the captured images) of the firstdirection of the captured image, the first direction corresponding tothe pitch direction or the yaw direction of the gimbal; determining thepitch angle and/or the yaw angle of the gimbal based on the deviationangle and the elevation angle and/or the horizontal angle. In someembodiments, the elevation angle and/or the horizontal angle of theimaging location may be directly set by the user. Assuming the imaginglocation information corresponding to the imaging location is (x, y, z),a directional angle may be defined by the imaging location pointing tothe target object. The elevation angle may be defined as arctan(z/x),the horizontal angle may be defined as arctan (y/x). the user settingthe elevation angle is to set the ratio between x and z. The usersetting the horizontal angle is to set the ratio between x and y. insome embodiments, determining the deviation angle of the target objectrelative to the center line of the first direction of the captured imagemay include: obtaining a first total pixel distance of the capturedimage in the first direction and the vertical field of view and/orhorizontal field of view of the imaging device; determining a firstdeviation pixel distance of the target object to the center line of thefirst direction of the captured image; determining the deviation angleof the target object relative to the center line of the first directionof the captured image based on the first total pixel distance, thevertical field of view and/or the horizontal field of view, and thefirst deviation pixel distance. It should be noted that when computingthe pitch angle of the gimbal, the deviation angle of the target objectrelative to the center line of the first direction of the captured imageis determined based on the vertical field of view, and when computingthe yaw angle of the gimbal, the deviation angle of the target objectrelative to the center line of the first direction of the captured imageis determined based on the horizontal field of view.

In some embodiments, referring to FIG. 5, after the user sets theexpected display location of the target object in the captured image,the first deviation pixel distance of the center of the target objectrelative to the X axis may be determined as d_rows. The user may set theelevation angle of the imaging location to be theta. The first totalpixel distance of the image (i.e., the height of the image) may be setas row_size, the vertical field of view of the imaging device may beVFOV, then the pitch angle “pitch” of the gimbal may be calculated as:

pitch=−theta-d_rows/row_size*VFOV;

or

pitch=−arctan(z/x)−d_row/row_size*VFOV.

In some embodiments, controlling the flight path of the UAV based on thelocation information and the flight mode may include: determining adistance between the target object and the imaging device; controllingthe flight path of the UAV based on the location information, the flightmode, and the distance between the target object and the imaging device,such that the height of the target object displayed in the capturedimage is a specified height, thereby satisfying a user demand for theimage composition. In some embodiments, determining the distance betweenthe target object and the imaging device may include: obtaining anactual height of the target object and a first total pixel distance ofthe captured image in the first direction; obtaining a pixel distancecorresponding to the actual height of the target object to be displayedin the first direction of the captured image, where the first directioncorresponds to the pitch direction of the gimbal; determining a distancebetween the target object and the imaging device based on the actualheight of the target object, the first total pixel distance, and thepixel distance corresponding to the actual height of the target objectin the first direction of the captured image. In some embodiments, asshown in FIG. 5, the actual height of the target object may be h, thedistance between the UAV and the target object may be defined as d(d=sqrt(x*x+y*y+z*z)), the elevation angle of the imaging location setby the user may be theta, the height of the image may be row_row, thevertical field of view of the imaging device may be VFOV. If during theimage composition, the user expects the height (i.e., the height in theY axis direction) of the target object to be displayed in the capturedimage to be tar_rows, then the distance d may satisfy:

cos(theta)*h/(2*d)=tan(tar_rows/(2*row_siz)*VFOV).

In some embodiments, if during image composition, the target object neednot be displayed at the horizontal (i.e., X axis) center of the capturedimage, then to accomplish image composition, after determining thedistance between the target object and the imaging device, the methodmay further include: obtaining the elevation angle of a predeterminedimaging location, a horizontal field of view of the imaging device, asecond total pixel distance of the captured image in a second direction,wherein the second direction corresponds to the yaw direction of thegimbal; determining a second pixel deviation distance of the targetobject relative to a center line of the second direction of the capturedimage; determining a moving distance in the pitch direction of thegimbal based on the second pixel deviation distance, the elevationangle, the horizontal field of view, the second total pixel distance,and the distance between the target object and the imaging device;controlling the attitude of the gimbal based on the moving distance inthe pitch direction of the gimbal. In some embodiments, as shown in FIG.5, the second pixel deviation distance of the target object relative tothe X axis may be d_col, the second total pixel distance of the image inthe X axis direction may be col_size, the horizontal field of view ofthe camera may be HFOV, the distance between the UAV and the target maybe d, the elevation angle of the imaging location set by the user may betheta, then the moving distance y in the pitch direction of the gimbalmay be:

y=sin(d_col/col_size*HFOV)*d*cos(theta).

In some embodiments, during the image composition, if the target objectneed not be displayed at the horizontal (i.e., X axis) center of thecaptured image, then to accomplish the desired image composition, theattitude of the gimbal may be controlled, including: obtaining ahorizontal field of view of the imaging device, a second total pixeldistance of the captured image in a second direction, where the seconddirection corresponds to the yaw direction of the gimbal; determining asecond pixel deviation distance of the target object to a center line ofthe second direction of the captured image; determining a yaw angle ofthe gimbal based on the second total pixel distance, the horizontalfield of view, and the second pixel deviation distance; controlling theattitude of the gimbal based on the yaw angle. In some embodiments, asshown in FIG. 5, the second pixel deviation distance of the targetobject relative to the X axis may be d_col, the second total pixeldistance of the image in the X axis direction may be col_size, thehorizontal field of view of the camera may be HFOV, then the yaw angleof the gimbal may be d_col/col_size*HFOV.

In some embodiments, the image composition may use the location in thecaptured image where the target object or the background identificationis to be displayed as the basis for image composition. In someembodiments, background identifications such as sky, building, or seasurface, etc., may be recognized through a classification algorithm suchas convolutional neural network (“CNN”), thereby achieving better imagecomposition.

In the above descriptions, the coordinate system XOY of the targetobject is used as a reference to deduce the location relationshipbetween the target object and the UAV. When the UAV is in flight, thelocation relationship between the target object and the UAV may bededuced through other methods. In some embodiments, the user operatingthe UAV may be the target object. The UAV may take off from the palm ofthe user. The location difference between the UAV and the user may beignored, and it may be assumed that the locations of the UAV and thetarget object are on a same straight line. In some embodiments, the useroperating the UAV may be the target object. The UAV may take off fromthe palm of the user by scanning the face of the user. The imagingdevice may be mounted at the front side of the aircraft body of the UAV.When scanning the face, the user may extend both arms, such that theimaging device faces the face of the user. The location relationshipbetween the UAV and the user may be deduced based on an assumed typicallength of the arm and a size of the human face.

In some embodiments, by setting the flight mode, the UAV mayautonomously fly based on the set flight mode and the locationinformation of the target object, thereby realizing relatively complexflight paths, especially the flight paths having a strong regularity. Byobtaining orientation information of the target object relative to theUAV through image recognition, the attitude of the gimbal may becontrolled, such that the target object is located in the capturedimage. According to the present disclosure, manual control by theoperator is not needed for realizing the control of the UAV and thegimbal. The captured images are smoother, and the image compositions arericher and more accurate.

In some embodiments, when the target object cannot be recognized throughimage recognition from the captured images, the orientation informationof the target object relative to the UAV may not be determined, whichresult in the UAV being unable to control the attitude of the gimbalbased on the orientation information to render the target object to belocated in the captured images. In order to continue to track and imagethe target object after the target object is lost, in one embodiment,the method may also include: when it is determined that the targetobject cannot be recognized in the image, then replacing the step ofcontrolling the attitude of the gimbal based on the orientationinformation by a step of controlling the attitude of the gimbal based onthe location information.

In some embodiments, controlling the flight path of the UAV based on thelocation information and the flight mode may include: controlling theUAV to move to a reset location. In some embodiments, after the UAVaccomplishes a flight based on the flight strategy of the flight mode,the UAV may automatically move to the reset location, such that the UAVis consistently located at the same takeoff location. The reset locationmay be a specific positioning coordinate location obtained by the UAVthrough GPS positioning. It should be noted that during the process ofthe UAV moving to the reset location, if the UAV receives a joystickoperation signal transmitted by an external device (e.g., a remotecontrol device that controls the operation of the UAV), the UAV mayimmediately terminate the current operation for moving to the resetlocation.

In some embodiments, the method may also include: if the joystickoperation signal transmitted by the external is received, controlling atleast one of the flight of the UAV or the attitude of the gimbal basedon the joystick operation signal. In this embodiment, the joystickoperation signal is generated through the user operating a remotecontrol device that controls the UAV. In some embodiments, the joystickoperation signal may include at least one of a signal controlling theUAV to ascend or descend perpendicular to the ground, a signalcontrolling the UAV to move away or closer to the target object, asignal for controlling the flight velocity of the UAV, a signal forcontrolling the yaw angle of the gimbal, a signal for controlling therotation of the aircraft body of the UAV, or a signal controlling otherUAV parameters or gimbal parameters.

Referring to FIG. 7, the remote control device that controls the UAV mayinclude two joysticks. Each joystick may include four adjustmentdirection for four degrees of freedom. One of the joysticks may includethe ascend/descend and left rotation/right rotation operations. Anotherjoystick may include the fore/aft and left/right operations. Theascend/descend correspond to the ascend/descend operations for theheight of the UAV. The left rotation/right rotation correspond to theyaw of the gimbal. The left/right correspond to the roll of the gimbal.The fore/aft correspond to the pitch of the gimbal.

In some embodiments, when the UAV is in a circling mode, the leftrotation/right rotation may be controlled, which respectively correspondto the left and right image compositions of the target object in thecaptured image, i.e., the left and right locations of the target objectin the captured image. The fore/aft controls respectively correspond tothe increase and decrease of the circling radius of the UAV circlingaround the target object. The ascend/descend controls respectivelycorrespond to the increase and decrease of the UAV height (in adirection perpendicular to the ground) while the UAV circles around thetarget object.

In some embodiments, when the UAV is in the slant line mode, the leftrotation/right rotation controls respectively correspond to the left andright image compositions of the target object in the captured image,i.e., the left and right locations of the target object in the capturedimage. The fore/aft controls correspond to the acceleration anddeceleration of the flight velocity of the UAV. The left/right andascend/descend controls are ineffective operations.

In some embodiments, when the UAV is in the roaring mode, the leftrotation/right rotation controls correspond to the rotation of theaircraft body of the UAV, and are configured to control the rotation ofthe aircraft body, thereby realizing the rotation of the lens of theimaging device, to obtain a rotating shot using the target object as acenter, further increasing the aesthetics of the captured image. Thefore/aft controls and the left/right controls are ineffectiveoperations. The ascend/descend controls respectively correspond to theacceleration and deceleration of the ascending velocity of the UAV.

In some embodiments, when the UAV is in the spiral mode, the leftrotation/right rotation controls respectively correspond to the left andright image compositions of the target object in the captured image,i.e., the left and right locations of the target object in the capturedimage. The fore/aft controls respectively correspond to the increase anddecrease of the spiral radius. The left/right controls respectivelycorrespond to the acceleration and deceleration of the transversalflight velocity (i.e., in the direction parallel with the ground) of thespiral flight. The ascend/descend controls respectively correspond tothe acceleration and deceleration of the spiral ascending velocity ofthe UAV, or the acceleration and deceleration of the spiral descendingvelocity of the UAV.

Embodiments of the present disclosure also provide an imaging controlmethod. The method may be implemented in a smart terminal 2 installedwith at least one application (“APP”). In some embodiments, the smartterminal may be communicatively connected with the UAV.

Referring to FIG. 8, the imaging method may include the following steps:

Step S801: receiving a user instruction;

where the user instruction may be directly input at the smart terminal 2by the user. In one embodiment, the smart terminal 2 may include an APPfor the user to input the user instruction. In some embodiments, the APPmay be configured to display the images transmitted back from the UAV.

In some embodiments, the user instruction may include: determining atarget object to be recognized. In some embodiments, after determiningthe target object to be recognized, the method may also include:recognizing feature information of the target object to be tracked inthe currently displayed image. The feature information may be apredetermined location or a predetermined size of the target object tobe displayed in the captured image. In some embodiments, the userinterface of the smart terminal may display in real time the imagecaptured by the imaging device at the current time instance. The usermay directly click the target object to be recognized on the image ofthe current time instance. The smart terminal may recognize the targetobject selected by the user based on image recognition technology toobtain feature information of the target object to be recognized. Thefeature information of the target object may be a predetermined locationof the target object, or a predetermined size of the target object, orother information such as the grayscale, texture, etc., which may makeit convenient for the subsequent tracking of the target object. In someembodiments, the method for the user to select the target object to berecognized may include: the user directly clicking an object in theimage of the current time instance displayed in the user interface ofthe smart terminal, then the object becomes the target object to berecognized. In some embodiments, the method for the user to select thetarget object to be recognized may include: the user uses a size frameto surround an object in the image of the current time instancedisplayed in the user interface of the smart terminal, then thesurrounded object becomes the target object to be recognized. In someembodiments, the size frame may be just enough to surround the targetobject to be recognized, or the size frame may be the minimum regulargraphic frame (e.g., the square frame or the circular frame) that cansurround the target object to be recognized. In some embodiments, thefeature information may include a predetermined location or apredetermined size of the target object in the captured image, therebyinstructing the UAV 1 to control the attitude of the gimbal such thatthe target object is consistently located at the predetermined locationin the captured images, and the size of the target object in thecaptured images is consistently the predetermined size, in order toobtain better image composition effects.

In some embodiments, the predetermined location of the target object inthe captured images is a predetermined location of a center location ofthe target object (which may be other locations of the target object) inthe image of the current time instance, when the user selects the targetobject to be recognized. The predetermined size of the target object inthe captured image is a value of the production of a pixel height and apixel width of the target object in an image captured at the currenttime instance. In some embodiments, in order to improve the aestheticsof the image composition of the captured images and to enhance therichness of the content of the captured images, the user instruction mayinclude: an expected display location of a specified location of abackground identification in the captured image. As a result,diversified image composition demands may be satisfied. The richness andaesthetics of the captured images may be enhanced. Specifically, in someembodiments, the background identification may include at least one of aground, a sky, a sea surface, a building, or other backgroundidentification.

In some embodiments, the user instruction may include: an elevationangle or a horizontal angle of an imaging location, which may be used tofurther determine a pitch angle or a yaw angle of a gimbal, therebyproviding better image compositions, such that the target object islocated at a predetermined location in the captured images.

In some embodiments, the user instruction may include: at least one of aflight distance and a flight velocity, which may be used to instruct theUAV to automatically accomplish a flight in each flight mode based onthe flight distance and/or the flight velocity. The flight distance andflight velocity corresponding to each flight mode may be set based onactual needs, thereby satisfying diversified demands of the user.

Step S802: generating a starting instruction based on the userinstruction. The starting instruction may include a flight mode of theUAV. The starting instruction may be configured to trigger theautonomous flight of the UAV based on the flight mode.

In some embodiments, the flight mode may be a default flight mode. Thedefault flight mode may be a predetermined flight mode or a combinationof multiple predetermined flight modes. Specifically, after the smartterminal 2 receives the user instruction (e.g., after the user pressesan operation button or inputs certain instruction information), thesmart terminal 2 may select the default flight mode and may generate thestarting instruction based on the default flight mode.

In some embodiments, the user instruction may include a mode selectioninstruction. The mode selection instruction may include a flight modefor instructing the flight of the UAV. In some embodiments, the user mayselect the flight mode of the UAV based on actual needs. Specifically,the smart terminal 2 may be pre-set with multiple flight modes for theuser to select. The user may select one or more flight modes from themultiple selectable flight modes provided by the smart terminal 2 basedon actual needs, thereby instructing the UAV to realize flights based ondifferent flight modes, to obtain captured images from different fieldof views.

In some embodiments, the flight mode may include at least one of a slantline mode, a circling mode, a spiral mode, a roaring mode, a cometcircling mode, or other flight mode (e.g., a straight line mode). Eachflight mode may include a corresponding flight strategy. The flightstrategy may be configured to instruct the flight of the UAV. The flightstrategy corresponding to each flight mode may refer to the aboverelated descriptions.

Step S803: transmitting the starting instruction to the UAV.

Step S804: receiving and storing a return video stream transmitted backby the UAV under the flight mode.

In each flight mode, the UAV may store, in real time, the current videodata (i.e., the raw data stream) captured by the imaging device, and maycompress the raw data stream in real time to generate a return videostream and transmit it back to the smart terminal 2, such that the smartterminal 2 can display, in real time, the images currently captured bythe UAV.

In step S804, after the smart terminal 2 receives the return videostream, the smart terminal 2 may cache the received return video stream,thereby obtaining a complete return video stream captured by the UAV inthe flight mode.

In some embodiments, the user may set the flight mode on the smartterminal 2, such that the UAV can autonomously fly based on the flightmode set by the user and the location information of the target object.As a result, the UAV may realize relatively complex flight paths,particularly the flight paths having a relatively strong regularity. TheUAV may obtain orientation information of the target object relative tothe UAV through image recognition, thereby controlling the attitude ofthe gimbal, such that the target object is located in the capturedimages. The control of the UAV and the gimbal may be realized withoutrequiring an operator to manually control the remote control device. Thecaptured images are smoother, and the image compositions are richer andmore accurate.

In some embodiments, in the field of UAV, the return video streamtransmitted by the UAV during a flight is generally provided to the userfor directly viewing. Because the size of the video stream transmittedto a ground based device (e.g., a smart terminal such as a smart phone,a tablet, etc.) from the UAV during the flight is typically large, theuser may have difficult in sharing the return video stream transmittedfrom the UAV in his/her social media platform, such as his/her friendcircle. Currently, in most situations, the user needs to manually editthe return video stream transmitted from the UAV to obtain a small videothat is more convenient for sharing. However, the manual editingperformed by the user to obtain the small video may not be professional,and the special effect of the small video may be poor. To address theabove issue, referring to FIG. 9, after step S804, the method may alsoinclude:

Step S901: processing the return video stream to a generate videofootage of a first predetermined duration. The first predeterminedduration may be shorter than a duration of the return video stream.

This embodiment does not need the manual editing of the user. Throughthe processing of the return video stream, the large return video streamcan be converted into a small video that is convenient for sharing(i.e., a video footage of the first predetermined duration). The usermay conveniently share the video in social media platforms, such ashis/her friend circle. The first predetermined duration may be set basedon actual needs, such as 10 seconds, which makes the small videoconvenient for sharing.

It should be noted that the small video of the present disclosure refersto a video having a duration shorter than a specified duration (whichmay be set based on actual needs). Of course, in some embodiments, thesmall video may refer to a video having a size or capacity smaller thana specified size or capacity (which may be set based on actual needs).

Step S901 may be executed after it is determined that the UAV satisfiesa specified condition.

In some embodiments, the specified condition may include: the UAVaccomplishes a flight in the flight mode. In this embodiment, at thesame time the UAV accomplishes the flight in the flight mode, the smartterminal 2 may receive a complete return video stream captured by theUAV in the flight mode, which makes it convenient for the user to selecta processing direction based on all or some of the information includedin the return video stream.

In some embodiments, the smart terminal 2 may determine, based on thereturn video stream transmitted back by the UAV, whether the UAVaccomplishes the flight in the flight mode. In some embodiments, the UAVmay add flight status information corresponding to the flight mode inthe images captured during the flight in the flight mode, and maycompress in real time the raw data stream having the flight statusinformation, and transmit the compressed raw data stream to the smartterminal 2. That is, the return video stream obtained by the smartterminal 2 may also include the flight status information. The smartterminal 2 may determine, based on the flight status informationincluded in the return video stream, whether the UAV accomplishes theflight in the flight mode. Specifically, if the smart terminal 2determines that the flight status information included in the returnvideo stream changes from the flight status information corresponding tothe flight mode to the flight status information corresponding toanother flight mode, or the return video stream changes from a returnvideo stream having the flight status information corresponding to theflight mode to a return video stream that does not have the flightstatus information, it may indicate that the UAV has accomplished theflight in the flight mode.

In some embodiments, after the UAV accomplishes the flight in the flightmode, the smart terminal 2 may receive information indicating thetermination of the flight mode from the UAV. As a result, the smartterminal 2 may determine that the UAV has accomplished the flight in theflight mode.

In some embodiments, the specified condition may include: receiving areturn video stream transmitted back from the UAV while the UAV flies inthe flight mode. In some embodiments, the smart terminal 2 may executestep S901 immediately after receiving the return video streamtransmitted from the UAV while the UAV flies in the flight mode, withoutwaiting for the UAV to complete the execution of the flight mode. Thismay save time spent for generating the small video. The smart terminal 2may generate the small video at the same time when the UAV completes theflight in the flight mode.

To reduce the size of the return video stream, and to generate the smallvideo that is convenient for sharing, in some embodiments, step S901 mayinclude: processing the return video stream with frame extraction togenerate a video footage of a first predetermined duration.

Specifically, in some embodiments, processing the return video streamwith frame extraction to generate the video footage of the firstpredetermined duration may include: processing the return video streamwith frame extraction based on at least one of the flight mode, a flightvelocity, or a flight direction of the UAV, to generate the videofootage of the first predetermined duration. By associating the smallvideo to be generated with at least one of the flight mode, the flightvelocity, or the flight direction of the UAV, a degree of fit betweenthe small video and the images captured by the UAV may be higher. Inaddition, the images included in the small video to be generated may bericher, and the image compositions may better match the flightparameters of the UAV.

In some embodiments, processing the return video stream with frameextraction to generate the video footage of the first predeterminedduration may include: processing the return video stream with frameextraction based on a duration of the return video stream and a numberof frames of the return video stream, to generate the video footage ofthe first predetermined duration. In this embodiment, while the size ofthe return video stream is reduced to generate the small video that isconvenient for sharing, a small video having a higher degree of fit withthe return video stream may be obtained based on the number of frames ofthe return video stream, thereby presenting the images captured by theUAV in a relatively complete manner.

In some embodiments, in order to present the images captured by the UAVin a relatively complete manner, processing the return video stream withframe extraction based on the duration of the return video stream andthe number of frames of the return video stream, to generate the videofootage of the first predetermined duration may include: dividing thereturn video stream into multiple segments to obtain multiple segmentsof the return video stream; processing a portion of the multiplesegments of the return video stream with frame extraction to obtainframe extraction images of the corresponding segments of the returnvideo stream; and generating the video footage of the firstpredetermined duration based on another portion of the return videostream and the frame extraction images of the corresponding segments ofthe return video stream.

In some embodiments, to reserve a beginning portion and an endingportion of the images captured by the UAV so as to ensure the integrityof the generated small video, dividing the return video stream intomultiple segments may include: dividing the return video stream into atleast three segments based on a sequential order in the imaging time.Processing a portion of the multiple segments of the return video streamwith frame extraction to obtain frame extraction images of thecorresponding segments of the return video stream may include:processing a segment of the return video stream whose imaging time islocated at a middle time segment among the at least three segments ofthe return video stream with frame extraction, to obtain the frameextraction images corresponding to this segment of the return videostream.

In some embodiments, to obtain a relatively smooth video footage,processing a portion of the multiple segments of the return video streamwith frame extraction to obtain frame extraction images of thecorresponding segments of the return video stream may include:processing a corresponding segment of the return video stream with frameextraction based on a predetermined frame extraction speed, to obtainthe frame extraction images of the corresponding segment of the returnvideo stream. In this embodiment, the frame extraction is performed onthe corresponding segment of the return video stream at a constantspeed, thereby avoiding discontinuity in the video footage that may becaused by non-constant frame extraction speed. In some embodiments, theframe extraction speeds for the multiple segments of the return videostream may be the same, which further improves the continuity of thegenerated video footage, thereby ensuring that the generate videofootage is relatively smooth.

In some embodiments, step S901 may include: compressing the return videostream to reduce a size of the return video stream, and to obtain avideo footage that is convenient for sharing.

In some embodiments, the method may also include: transmitting the videofootage to a remote terminal server, thereby realizing sharing of thesmall video. The remote terminal server may be a third-party website,such as video sharing websites Youku™, Tudou™, etc., or social medianetworks, such as friend circle. In some embodiments, the step oftransmitting the video footage to the remote terminal server may beexecuted after step S901 is completed, thereby realizing quick sharingof the small video. In some embodiments, prior to transmitting the videofootage to the remote terminal server, the method also includes:receiving a sharing instruction input by a user. The sharing instructionmay include the corresponding remote terminal server. The video footagemay be transmitted to the remote terminal server based on the sharinginstruction, thereby realizing flexible sharing of the small video basedon actual needs of the user.

In some embodiments, during a flight of the UAV, if the signal of thetransmission link between the UAV and the smart terminal 2 is poor, thequality of the return video stream transmitted back by the UAV to thesmart terminal 2 through an image transmission method may also be poor.Correspondingly, the quality of the generated small video may also bepoor.

With regard the issue of poor quality of the return video stream,referring to FIG. 10, in the present disclosure, the video footagegenerating method may also include the following steps:

Step S1001: obtaining a raw data stream captured by the UAV.

In each flight mode, the UAV may store, in real time, the current videodata (i.e., the raw data stream) captured by the imaging device, and maycompress the raw data stream in real time to generate a return videostream and transmit it back to the smart terminal 2 through an imagetransmission method, such that the smart terminal 2 can display, in realtime, the images currently captured by the UAV. To improve the qualityof the generated small video, the smart terminal 2 may obtain the rawdata stream stored by the UAV, and generate the small video based on theraw data stream.

In some embodiments, the raw data stream captured by the UAV may bestored in the UAV or a storage unit of the imaging device. In someembodiments, the smart terminal 2 may directly retrieve the raw datastream stored in the storage unit that is captured by the UAV. It shouldbe noted, that step S1001 and step S901 differ in the data transmissionmethods: in step S901, the UAV transmits the captured video stream tothe smart terminal 2 through a wireless communication method. Becausethe communication distance between the UAV and the smart terminal 2 isfar, it is possible that the communication quality between the UAV andthe smart terminal 2 is poor; in step S1001, the smart terminal 2 mayretrieve the raw data stream from the storage unit through a wiredcommunication method, or under the condition of maintaining relativelygood wireless communication quality, the smart terminal 2 may directlyretrieve the raw data stream from the storage unit, thereby ensuringthat the smart terminal 2 can obtain the raw data stream having imagesof good quality. In some embodiments, the storage unit may be acomponent capable of storing data, such as a secure digital (“SD”) cardor a hard disk or a magnetic disk.

In some embodiments, in the video data stored in the storage unit, theraw data stream may include a corresponding video label. The smartterminal 2 may search and find the corresponding raw data stream basedon the label from the storage unit. Specifically, step S1001 may beexecuted after the UAV satisfies a specified condition. The specifiedcondition may include: the UAV accomplishes a flight in a flight mode.Specifically, after the UAV accomplishes the flight in the flight mode,the smart terminal 2 may directly retrieve the raw data stream capturedby the UAV and stored in the storage unit, thereby obtaining a raw datastream having images of good quality, and processing the raw data streamto generate a small video having images of good quality.

Step S1002: determining a raw video stream captured by the UAV in theflight mode based on the raw data stream.

In some embodiments, step S1002 may include: determining the raw videostream captured by the UAV in the flight mode from the raw data streambased on the video stream label corresponding to the flight mode.Through the video label, the raw video stream captured by the UAV in theflight mode may be relatively accurately and quickly obtained from alarge amount of video streams, thereby more quickly generating the smallvideo in the flight mode.

Step S1003: processing the raw video stream to generate a new videofootage of a second predetermined duration. The second predeterminedduration may be shorter than a duration of the raw video stream.

In some embodiments, step S1001, step S1002, and step S1003 may beexecuted after it is determined that a resolution of the video footageobtained based on the return video stream is lower than a predeterminedresolution, thereby obtaining the new video footage having relativelyhigh quality.

In some embodiments, the method may also include: transmitting the newvideo footage to the remote terminal server, thereby realizing sharingof the small video. In some embodiments, the remote terminal server maybe a third-party website, such as video sharing websites Youku™, Tudou™,etc., or social media networks, such as friend circle. In someembodiments, transmitting the video footage to the remote terminalserver may be executed after step S1003 is complete, thereby realizingquick sharing of the small video. In some embodiments, prior totransmitting the new video footage to the remote terminal server, themethod may include: receiving a sharing instruction input by a user. Thesharing instruction may include the corresponding remote terminalserver. The video footage may be transmitted to the remote terminalserver based on the sharing instruction, thereby realizing flexiblesharing of the small video based on the actual needs of the user.

In some embodiments, the smart terminal 2 may simultaneously executestep S1001, step S1002, step S1003, and step S804 and step S901, therebyobtaining two video footages for the user to select, which increases therichness of the selection.

In some embodiments, the method may also include: based on the videofootage generated in step S901 and the new video footage generated instep S1003, at least one of the two video footages is transmitted to theremote terminal server. In some embodiments, one of the video footagegenerated in step S901 and the new video footage generated in step S1003that has a higher resolution may be transmitted to the remote terminalserver. In some embodiments, based on the video footage generated instep S901 and the new video footage generated in step S1003, before atleast one of the two video footages is transmitted to the remoteterminal server, the method may also include: receiving a sharinginstruction input by a user. The sharing instruction may include theremote terminal server and an identification of a video to be shared.The identification of the video to be shared may be a descriptioncorresponding to at least one of the video footage generated in stepS901 and the new video footage generated in step S1003. The method mayalso include transmitting one of the video footage generated in stepS901 and the new video footage generated in step S1003 to the remoteterminal server based on the sharing instruction, thereby realizingflexible sharing of the small video based on the actual needs of theuser.

In some embodiments, the second predetermined duration may be set basedon actual needs. In some embodiments, the second predetermined durationmay be set to be the same as the first predetermined duration.

In some embodiments, the strategy used for processing the raw videostream in step S1003 may be similar to the strategy used for processingthe return video stream in step S802, the details of which may refer tothe strategy used for processing the return video stream in step S901,which is not repeated here.

Steps of the method that have not been described in great detail canrefer to the same or similar steps of the imaging control methoddescribed previously, which are not repeated here.

In some embodiments, corresponding to the imaging control methodsdescribed above, the present disclosure also provides an imaging controldevice. The device may be implemented at the UAV 1.

Referring to FIG. 11, the imaging control device may include a firstprocessor 11. The first processor 11 may be configured to execute stepsof the imaging control method described above.

In some embodiments, the first processor 11 may be communicativelyconnected with the smart terminal 2. Thus, the first processor 11 mayreceive a starting instruction from the smart terminal 2, and maytransmit images captured by the UAV and other data information of theUAV to the smart terminal 2.

In some embodiments, the first processor 11 may be configured as acontroller in a dedicated control device, or may be configured as aflight controller of the UAV, or may be configured as a gimbalcontroller.

Portions of the device that function in the same or similar manner asthose described in the above disclosed imaging control methods may referto the corresponding descriptions of the methods, which are not repeatedhere.

In some embodiments, corresponding to the imaging control methoddisclosed herein, the present disclosure provides another imagingcontrol device. The device may be implemented in the smart terminal 2that has installed with an APP.

Referring to FIG. 11, the imaging control device may include a secondprocessor 21. The second processor 21 may be configured to execute thesteps of the above disclosed imaging control methods.

In some embodiments, the second processor 21 may be communicativelyconnected with a control device of the UAV 1. The control device of theUAV 1 may be realized by a dedicated control device, or may be realizedby a flight controller of the UAV, or may be realized by a gimbalcontroller. The second processor 21 may be configured to transmit thestarting instruction to the UAV 1 to instruct the UAV to perform aerialphotography. The second processor 21 may also be configured to receiveimages captured by the UAV or other data information of the UAV.

Portions of the device that function in the same or similar manner asthose described in the above disclosed imaging control methods may referto the corresponding descriptions of the methods, which are not repeatedhere.

In some embodiments, corresponding to the imaging control methodsdisclosed herein, the present disclosure provides another imagingcontrol device. The device may be implemented at the UAV 1.

Referring to FIG. 12, the device may include:

a first receiving module 101 configured to receive the startinginstruction, where the starting instruction includes a flight mode ofthe UAV;

a first control module 102 configured to control the UAV to autonomouslyfly based on the flight mode;

a location computing module 103 configured to obtain locationinformation of the target object in the flight mode, and to obtainorientation information of the target object relative to the UAV basedon the target object recognized from the image captured by the imagingdevice;

a second control module 104 configured to control a flight path of theUAV based on the location information and the flight mode;

a third control module 105 configured to control attitude of a gimbalbased on the orientation information to render the target object toappear in the image captured by the imaging device.

In some embodiments, referring to FIG. 13, the device may also includean imaging control module 106 configured to control the imaging deviceto record videos in the flight mode, and to transmit the video data tothe smart terminal 2.

In some embodiments, as shown in FIG. 13, the device may also include afirst determination module 107. When the first determination module 107determines that the location computing module 103 cannot recognize thetarget object in the image, the third control module 105 may replace thestep of controlling the attitude of the gimbal based on the orientationinformation with the step of controlling the attitude of the gimbalbased on the location information.

In some embodiments, controlling the attitude of the gimbal may include:controlling at least one of the pitch angle, the yaw angle, or the rollangle of the gimbal.

In some embodiments, the location computing module 103 may be configuredto determine the pitch angle and/or the yaw angle of the gimbal.

In some embodiments, the process for determining the pitch angle and/orthe yaw angle of the gimbal may include: determining at least one of thepitch angle or the yaw angle of the gimbal based on an expected displaylocation of a specified location of a background identification to bedisplayed in the captured image.

In some embodiments, determining at least one of the pitch angle or theyaw angle of the gimbal based on the expected display location of thespecified location of the background identification to be displayed inthe captured image may include:

obtaining a first total pixel distance of a captured image in a firstdirection and a pixel distance of the expected display location of thespecified location of the background identification to be displayed inthe captured image to an edge of the image in the first direction, wherethe first direction may correspond to the pitch direction or the yawdirection of the gimbal;

determining the pitch angle and/or the yaw angle of the gimbal based onthe first total pixel distance, the pixel distance, and the size of thevertical field of view or the horizontal field of view of the imagingdevice.

In some embodiments, the background identification may include at leastone of a ground, a sky, a sea surface, or a building.

In some embodiments, the process of determining the pitch angle and/orthe yaw angle of the gimbal may include:

obtaining an elevation angle and/or a horizontal angle of apredetermined imaging location;

determining a deviation angle of the target object relative to a centerline of the first direction of the captured image, the first directioncorresponding to the pitch direction or the yaw direction of the gimbal;

determining the pitch angle and/or the yaw angle of the gimbal based onthe deviation angle and the elevation angle and/or the horizontal angle.

In some embodiments, determining a deviation angle of the target objectrelative to the center line of the first direction of the captured imagemay include:

obtaining a first total pixel distance of the captured image in thefirst direction and a vertical field of view and/or horizontal field ofview of the imaging device;

determining a first deviation pixel distance of the target object to thecenter line of the first direction of the captured image;

determining the deviation angle of the target object relative to thecenter line of the first direction of the captured image based on thefirst total pixel distance, the vertical field of view and/or thehorizontal field of view, and the first deviation pixel distance.

In some embodiments, controlling the flight path of the UAV based on thelocation information and the flight mode may include:

determining a distance between the target object and the imaging device;

controlling the flight path of the UAV based on the locationinformation, the flight mode, and the distance between the target objectand the imaging device.

In some embodiments, determining the distance between the target objectand the imaging device may include:

obtaining an actual height of the target object and a first total pixeldistance of the captured image in the first direction;

obtaining a pixel distance corresponding to the actual height of thetarget object to be displayed in the first direction of the capturedimage, where the first direction corresponds to the pitch direction ofthe gimbal;

determining a distance between the target object and the imaging devicebased on the actual height of the target object, the first total pixeldistance, and the pixel distance corresponding to the actual height ofthe target object in the first direction of the captured image.

In some embodiments, the location computing module 103 may be configuredto obtain the elevation angle of the predetermined imaging location, thehorizontal field of view of the imaging device, and a second total pixeldistance of the captured image in a second direction, after determiningthe distance between the target object and the imaging device, whereinthe second direction may correspond to the yaw direction of the gimbal;to determine a second pixel deviation distance of the target objectrelative to a center line of the second direction of the captured image;to determine a moving distance in the pitch direction of the gimbalbased on the second pixel deviation distance, the elevation angle, thehorizontal field of view, the second total pixel distance, and thedistance between the target object and the imaging device. The thirdcontrol module 105 may be configured to control the attitude of thegimbal based on the moving distance in the pitch direction of thegimbal.

In some embodiments, the location computing module 103 may be configuredto obtain a horizontal field of view of the imaging device, a secondtotal pixel distance of the captured image in a second direction, wherethe second direction corresponds to the yaw direction of the gimbal; todetermine a second pixel deviation distance of the target object to acenter line of the second direction of the captured image; to determinea yaw angle of the gimbal based on the second total pixel distance, thehorizontal field of view, and the second pixel deviation distance. Thethird control module 105 may be configured to control the attitude ofthe gimbal based on the yaw angle.

In some embodiments, the gimbal and the UAV may be fixed relative to oneanother in a heading axis. The third control module 105 may beconfigured to control the pitch angle and/or the roll angle of thegimbal, and to control a heading angle of the UAV to control the yawangle of the gimbal.

In some embodiments, the flight mode may include at least one of thefollowing: a slant line mode, a circling mode, a spiral mode, a roaringmode, or a comet circling mode. Each flight mode may include acorresponding flight strategy. The flight strategy may be configured toinstruct the flight of the UAV.

In some embodiments, the flight strategy corresponding to the slant linemode may include: controlling, by the second control module 104, basedon the location information, the UAV to first fly along a horizontalplane and then to fly along a plane forming a certain angle with thehorizontal plane.

In some embodiments, the second control module 104 controlling the UAVto fly along the horizontal plane and then to fly along the planeforming the certain angle with the horizontal plane may include:controlling the UAV to fly along the horizontal plane; when it isdetermined that an angle between a line connecting the lowest point ofthe target object and a center of the UAV and a line connecting thehighest point of the target object and the center of the UAV is smallerthan a predetermined multiple of the field of view of the imagingdevice, then based on the location information, controlling the UAV tofly along the plane forming the certain angle with the horizontal plane.The predetermined multiple may be <1.

In some embodiments, the second control module 104 controlling the UAVto fly along the plane forming the certain angle with the horizontalplane may include: controlling the UAV to fly along a line connectingthe target object and the UAV in a direction away from the targetobject.

In some embodiments, the flight strategy corresponding to the slant linemode may include: controlling, by the second control module 104, the UAVto fly in an S shape curve away from the target object based on thelocation information.

In some embodiments, the flight strategy corresponding to the circlingmode may include: based on the location information, controlling, by thesecond control module 104, the UAV to fly circling around the targetobject based on a specified distance.

In some embodiments, the specified distance may be a default distance,or the specified distance may be distance information input by the user,or the specified distance may be a distance between the UAV and thetarget object at the current time instance.

In some embodiments, the flight strategy corresponding to the spiralmode may include: based on the location information, controlling, by thesecond control module 104, the UAV to fly circling around the targetobject in a flight path having a shape of a Fibonacci spiral, an equalratio spiral, an equiangular spiral, or an Archimedean spiral.

In some embodiments, the flight strategy corresponding to the spiralmode may also include: based on the location information, whilecontrolling, by the second control module 104, the UAV to fly circlingaround the target object in a flight path having a shape of a Fibonaccispiral, an equal ratio spiral, an equiangular spiral, or an Archimedeanspiral, in the meantime, also controlling, by the second control module104, the UAV to ascend or descend perpendicular to the ground based on apredetermined speed.

In some embodiments, the flight strategy corresponding to the roaringmode may include: based on the location information, controlling, by thesecond control module 104, the UAV to fly slantly according to apredetermined angle to a first specified location relative to the targetobject, and then controlling the UAV to ascend perpendicular to theground.

In some embodiments, the flight strategy corresponding to the cometcircling mode may include: based on the location information,controlling, by the second control module 104, the UAV to fly to asecond specified location in a direction moving closer to the targetobject, and after circling around the target object starting from thesecond specified location, fly away from the target object.

In some embodiments, each flight mode may include at least one of acorresponding flight path or a flight velocity.

In some embodiments, obtaining by the location computing module 103 thelocation information may include:

obtaining an information set including at least two groups of imaginginformation, the imaging information may include: imaging locationinformation and imaging angle information when the target object iscaptured; and

determining the location information of the target object based on atleast two groups of imaging information selected from the informationset, where locations corresponding to imaging location informationincluded in each selected group of imaging information may be different.

In some embodiments, determining the location information of the targetobject based on at least two groups of imaging information selected fromthe information set may include:

determining location initial estimation information of at least twotarget objects based on at least three groups of imaging information;and determining location information of each target object based on thelocation initial estimation information.

In some embodiments, the imaging location information may be thepositioning information of the UAV.

In some embodiments, obtaining, by the location computing module 103,the location information of the target object may include: obtainingpositioning information of the smart terminal 2. The smart terminal 2may be a terminal communicating with the UAV. The location informationmay be the positioning information.

In some embodiments, obtaining, by the location computing module 103,the orientation information of the target object relative to the UAVbased on the images captured by the imaging device may include:obtaining feature information of the target object to be tracked;recognizing the target object from the captured images based on imagerecognition technology and based on the feature information, andobtaining the orientation information of the target object relative tothe UAV.

In some embodiments, referring to FIG. 13, the device may also include areset module 108 configured to control the UAV to move to a resetlocation after the second control module 104 controls the flight path ofthe UAV based on the location information and the flight mode.

In some embodiments, referring to FIG. 13, the device may also include afourth control module 109 configured to control at least one of theflight of the UAV or the attitude of the gimbal based on a joystickoperation signal, after the first determination module 107 determinesthat the first receiving module 101 receives the joystick operationsignal transmitted from an external device.

In some embodiments, the joystick operation signal may include at leastone of a signal controlling the UAV to ascend or descend perpendicularto the ground, a signal controlling the UAV to move away or closer tothe target object, a signal for controlling the flight velocity of theUAV, a signal for controlling the yaw angle of the gimbal, or a signalfor controlling the rotation of the aircraft body of the UAV.

Portions of the device that function in the same or similar manner asthose described in the above disclosed imaging control methods may referto the corresponding descriptions of the methods, which are not repeatedhere.

In some embodiments, corresponding to the imaging control methodsdisclosed herein, the present disclosure also provides another imagingcontrol device. The device may be implemented in the smart terminal 2installed with an APP.

Referring to FIG. 14, the device may include:

a second receiving module 201 configured to receive a user instruction;

an instruction generating module 202 configured to generate a startinginstruction based on the user instruction. The starting instruction mayinclude a flight mode of the UAV. The starting instruction may beconfigured to trigger the UAV to autonomously fly based on the flightmode;

a transmitting module 203 configured to transmit the startinginstruction to the UAV, where the starting instruction is configured totrigger the UAV to autonomously fly based on the flight mode.

After the transmitting module 203 transmits the starting instruction tothe UAV, the second receiving module 201 may receive and store a returnvideo stream transmitted back by the UAV in the flight mode.

In some embodiments, the user instruction may include: determining atarget object to be tracked.

In some embodiments, the method may include: recognizing featureinformation of the target object to be tracked in the currentlydisplayed image, the feature information being a predetermined locationor a predetermined size of the target object to be displayed in thecaptured image.

In some embodiments, the user instruction may include: an expecteddisplay location of the specified location of the backgroundidentification to be displayed in the captured image. The backgroundidentification may include at least one of a ground, a sky, a seasurface, or a building.

In some embodiments, the use instruction may include: an elevation angleor a horizontal angle of an imaging location.

In some embodiments, the user instruction may include: at least one ofthe flight distance or the flight velocity of the UAV.

In some embodiments, the flight mode may be a default flight mode; orthe user instruction includes a mode selection instruction. The modeselection instruction includes a flight mode for instructing the flightof the UAV.

In some embodiments, the flight mode may include at least one of thefollowing: a slant line mode, a circling mode, a spiral mode, a roaringmode, or a comet circling mode. Each flight mode may include acorresponding flight strategy. The flight strategy may be configured toinstruct the flight of the UAV.

In some embodiments, referring to FIG. 15, the device may also include aprocessing module 204 configured to process the return video stream togenerate a video footage of a first predetermined duration. The firstpredetermined duration may be shorter than the duration of the returnvideo stream.

In some embodiments, referring to FIG. 15, the device may also include asecond determination module 105. The step of processing, by theprocessing module 204, the return video stream and generating the videofootage may be executed after the second determination module 205determines that the UAV satisfies a specified condition.

In some embodiments, the specified condition may include: the seconddetermination module 205 determines that the UAV has accomplished theflight in the flight mode.

In some embodiments, processing, by the processing module 204, thereturn video stream to generate the video footage of the firstpredetermined duration may include: processing the return video streamwith frame extraction to generate the video footage of the firstpredetermined duration.

In some embodiments, processing, by the second processing module 204,the return video stream with frame extraction to generate the videofootage of the first predetermined duration may include: processing thereturn video stream with frame extraction based on at least one of theflight mode, a flight velocity, or a flight direction of the UAV, togenerate the video footage of the first predetermined duration.

In some embodiments, processing, by the second processing module 204,the return video stream with frame extraction to generate the videofootage of the first predetermined duration may include:

processing the return video stream with frame extraction based on aduration of the return video stream and a number of frames of the returnvideo stream, to generate the video footage of the first predeterminedduration.

In some embodiments, processing, by the processing module 204, thereturn video stream with frame extraction based on the duration of thereturn video stream and the number of frames of the return video stream,to generate the video footage of the first predetermined duration mayinclude: dividing the return video stream into multiple segments toobtain multiple segments of the return video stream; processing aportion of the multiple segments of the return video stream with frameextraction to obtain frame extraction images of the correspondingsegments of the return video stream; and generating the video footage ofthe first predetermined duration based on another portion of the returnvideo stream and the frame extraction images of the correspondingsegments of the return video stream.

In some embodiments, dividing, by the processing module 204, the returnvideo stream into multiple segments may include: dividing the returnvideo stream into at least three segments based on a sequential order inthe imaging time. Processing, by the processing module 204, a portion ofthe multiple segments of the return video stream with frame extractionto obtain frame extraction images of the corresponding segments of thereturn video stream may include: processing a segment of the returnvideo stream whose imaging time is located at a middle time segmentamong the at least three segments of the return video stream with frameextraction, to obtain the frame extraction images corresponding to thissegment of the return video stream.

In some embodiments, processing, by the processing module 204, a portionof the multiple segments of the return video stream with frameextraction to obtain frame extraction images of the correspondingsegments of the return video stream may include: processing acorresponding segment of the return video stream with frame extractionbased on a predetermined frame extraction speed, to obtain the frameextraction images of the corresponding segment of the return videostream.

In some embodiments, referring to FIG. 15, the device may also include aretrieving module 206 and a determination module 207. The retrievingmodule 206 may be configured to obtain a raw data stream captured by theUAV. The determination module 207 may be configured to determine, basedon the raw data stream, a raw video stream captured by the UAV in theflight mode. The processing module 204 may be configured to process theraw video stream to generate a new video footage of a secondpredetermined duration. The second predetermined duration may be shorterthan the duration of the return video stream.

In some embodiments, determining, by the determination module 207, basedon the raw data stream, the raw video stream captured by the UAV in theflight mode may include: determining the raw video stream captured bythe UAV in the flight mode from the raw data stream based on a videostream label corresponding to the flight mode.

In some embodiments, the step of obtaining, by the retrieving module206, the raw data stream captured by the UAV may be executed after thesecond determination module 205 determines that the resolution of thevideo footage obtained based on the return video stream is lower thanthe predetermined resolution.

In some embodiments, referring to FIG. 15, the device may also include asharing module 208 configured to transmit at least one of the videofootage and the new video footage to the remote terminal server.

In some embodiments, prior to the sharing module 208 transmits at leastone of the video footage and the new video footage to the remoteterminal server, the second receiving module 201 may receive a sharinginstruction input by the user. The sharing instruction may include theremote terminal server and an identification of the video to be shared.The identification of the video to be shared may be an identificationcorresponding to at least one of the video footage and the new videofootage. The sharing module 208 may transmit at least one of the videofootage and the new video footage to the remote terminal server based onthe sharing instruction.

Portions of the device that function in the same or similar manner asthose described in the above disclosed imaging control methods may referto the corresponding descriptions of the methods, which are not repeatedhere.

In some embodiments, the present disclosure provides a non-transitorycomputer-readable storage medium. The computer-readable storage mediummay be configured to store program instructions, which when executed bya processor, cause the processor to perform the methods (e.g., theimaging control methods) disclosed herein.

In the present description, descriptions of reference terms such as “anembodiment,” “some embodiments,” “illustrative embodiment,” “example,”“specific example,” or “some examples,” mean that characteristics,structures, materials, or features described in relation to theembodiment or example are included in at least one embodiment or exampleof the present disclosure. In the present description, illustrativeexpression of the above terms does not necessarily mean the sameembodiment or example. Further, specific characteristics, structures,materials, or features may be combined in one or multiple embodiments orexamples in a suitable manner.

A process or method shown in a flow chart or described in any other formmay represent one or more module, segments, or parts ofcomputer-executable codes for realizing specific logical functions orfor executing specific steps. Other implementations of the disclosedmethods or functions may also be included in the present disclosure.Steps of the processes do not necessarily have to be executed in theorder shown in the flow chart or as described. Other orders orsequences, such as simultaneous execution or execution in reverse ordermay be adopted based on the functions to be realized. A person havingordinary skills in the art can understand that the present disclosure isnot limited to the illustrative order of the steps.

The logic or steps shown in the flow chart or otherwise described in thespecification may be regarded as representation of a list ofcomputer-executable codes for realizing certain logic functions. Thecomputer-executable codes may be embedded or encoded in acomputer-readable storage medium. The computer-readable storage mediummay be used by code-executing system, apparatus, or device (e.g., acomputer-based system, a system having a processing device or aprocessor, or other systems that can read and execute codes from acode-executing system, apparatus, or device). The computer-readablemedium may be used in combination with the code-executing system,apparatus, or device. In the present disclosure, the term“computer-readable medium” refers to a non-transitory device that mayinclude, store, communicate, broadcast, or transmit computer programcode for the code-executing system, apparatus, or device to execute, ormay be any device that may be used with the code-executing system,apparatus, or device. The non-transitory computer-readable storagemedium may include one or more of: an electrical connector having one ormore wiring layouts (e.g., electronic device, a portable computer diskcase (e.g., magnetic device), a random access memory (“RAM”), aread-only memory (“ROM”), an Electrically Programmable read only memory(“EPROM” or a flash memory), an optical device, or a Compact Disc-ROM(“CD-ROM”). In some embodiments, the computer-readable medium mayinclude a paper or other suitable medium printed with a computerprogram. The paper or other suitable medium may be optically scanned,edited, interpreted, or processed using other methods to obtain thecomputer program electronically, which may be stored in a computerstorage medium.

A person having ordinary skills in the art can appreciate that part orall of the above disclosed methods and processes may be implementedusing related electrical hardware, computer software, firmware, or acombination thereof. In the above embodiments, multiple steps or methodsmay be realized using software or firmware stored in thecomputer-readable storage medium and executable by a suitablecode-executing system. For example, if the disclosed methods andprocesses are implemented using hardware, the hardware may include atleast one of the following: a discrete logic circuit having a logic gatecircuit that may be configured to perform logic functions for digitalsignals, an application specific integrated circuit having suitablecombinations of logic gate circuits, a programmable gate array (“PGA”),a field programmable gate array (“FPGA”), etc.

A person having ordinary skills in the art can appreciate that some orall of the steps of the above disclosed methods may be implemented usingprograms instructing related hardware. The program may be stored in acomputer-readable storage medium. When the program is executed, one ofthe steps of the disclosed methods or a combination thereof may beperformed.

Various functional units or components may be integrated in a singleprocessing unit, or may exist as separate physical units or components.In some embodiments, two or more units or components may be integratedin a single unit or component. The integrated unit may be realized usinghardware, software functional modules, or a combination of hardware andsoftware functional modules. If the integrated units are realized assoftware functional units and sold or used as independent products, theintegrated units may be stored in a computer-readable storage medium.

The storage medium mentioned above is a read-only memory, a magneticdisk, or an optical disc, etc. Although various embodiments areillustrated and described, it can be understood that these embodimentsare illustrative only, and cannot be understood as limiting the scope ofthe present disclosure. A person having ordinary skills in the art canchange, modify, replace, or vary these embodiments within the scope ofthe present disclosure.

What is claimed is:
 1. An imaging control method, comprising: receivinga starting instruction comprising a flight mode of an unmanned aerialvehicle (“UAV”); controlling the UAV to fly autonomously based on theflight mode; obtaining, in the flight mode, location information of atarget object, and obtaining orientation information of the targetobject relative to the UAV based on the target object recognized from animage captured by an imaging device carried by a gimbal mounted on theUAV; controlling a flight path of the UAV based on the locationinformation and the flight mode; controlling an attitude of the gimbalto render the target object to appear in the image captured by theimaging device; and controlling the imaging device to record a video inthe flight mode, and to transmit video data to a terminal.
 2. Theimaging control method of claim 1, wherein the flight mode comprises atleast one of a slant line mode, a circling mode, a spiral mode, aroaring mode, or a comet circling mode, and wherein each flight modecomprises a corresponding flight strategy configured to instruct aflight of the UAV.
 3. The imaging control method of claim 2, wherein aflight strategy corresponding to the slant line mode comprises:controlling, based on the location information, the UAV to first flyalong a horizontal plane and then to fly along a plane forming a certainangle with the horizontal plane.
 4. The imaging control method of claim3, wherein controlling the UAV to first fly along the horizontal planeand then to fly along the plane forming the certain angle with thehorizontal plane comprises: controlling the UAV to fly along thehorizontal plane; and when it is determined that an angle between a lineconnecting a lowest point of the target object and a center of the UAVand a line connecting a highest point of the target object and thecenter of the UAV is smaller than a predetermined multiple of a field ofview of the imaging device, then based on the location information,controlling the UAV to fly along the plane forming the certain anglewith the horizontal plane, wherein the predetermined multiple may besmaller than
 1. 5. The imaging control method of claim 4, whereincontrolling the UAV to fly along the plane forming the certain anglewith the horizontal plane comprises: controlling the UAV to fly along aline connecting the target object and the UAV in a direction away fromthe target object.
 6. The imaging control method of claim 2, wherein aflight strategy corresponding to the slant line mode comprises:controlling the UAV to fly in an S shape curve away from the targetobject based on the location information, or wherein a flight strategycorresponding to the roaring mode comprises: controlling, based on thelocation information, the UAV to fly slantly according to apredetermined angle to a first specified location relative to the targetobject, and then controlling the UAV to ascend perpendicular to theground, or wherein a flight strategy corresponding to the comet circlingmode comprises: controlling, based on the location information, the UAVto fly to a second specified location in a direction moving closer tothe target object, and after circling around the target object startingfrom the second specified location, fly away from the target object. 7.The imaging control method of claim 2, wherein a flight strategycorresponding to the circling mode comprises: controlling, based on thelocation information, the UAV to fly circling around the target objectbased on a specified distance.
 8. The imaging control method of claim 7,wherein the specified distance is a default distance, or the specifieddistance is distance information input by the user, or the specifieddistance is a distance between the UAV and the target object at acurrent time instance.
 9. The imaging control method of claim 2, whereinthe flight strategy corresponding to the spiral mode comprises:controlling, based on the location information, the UAV to fly circlingaround the target object in a flight path having a shape of a Fibonaccispiral, an equal ratio spiral, an equiangular spiral, or an Archimedeanspiral.
 10. The imaging control method of claim 9, wherein a flightstrategy corresponding to the spiral mode comprises: while controlling,based on the location information, the UAV to fly circling around thetarget object in the flight path having the shape of the Fibonaccispiral, the equal ratio spiral, the equiangular spiral, or theArchimedean spiral, also controlling the UAV to ascend or descendperpendicular to the ground based on a predetermined speed.
 11. Theimaging control method of claim 2, wherein each flight mode comprises atleast one of a corresponding flight distance or a flight velocity. 12.An imaging control device, comprising: a memory configured to storecomputer-executable instructions; and a processor configured to retrieveand execute the computer-executable instructions to perform a methodcomprising: receiving a starting instruction comprising a flight mode ofan unmanned aerial vehicle (“UAV”); controlling the UAV to flyautonomously based on the flight mode; obtaining, in the flight mode,location information of a target object, and obtaining orientationinformation of the target object relative to the UAV based on the targetobject recognized from an image captured by an imaging device carried bya gimbal mounted on the UAV; controlling a flight path of the UAV basedon the location information and the flight mode; controlling an attitudeof the gimbal to render the target object to appear in the imagecaptured by the imaging device; and controlling the imaging device torecord a video in the flight mode, and to transmit video data to aterminal.
 13. The imaging control device of claim 12, wherein the flightmode comprises at least one of a slant line mode, a circling mode, aspiral mode, a roaring mode, or a comet circling mode, and wherein eachflight mode comprises a corresponding flight strategy configured toinstruct a flight of the UAV, or wherein a flight strategy correspondingto the slant line mode comprises: controlling, based on the locationinformation, the UAV to first fly along a horizontal plane and then tofly along a plane forming a certain angle with the horizontal plane. 14.The imaging control device of claim 13, wherein a flight strategycorresponding to the slant line mode comprises: controlling the UAV tofly in an S shape curve away from the target object based on thelocation information, or wherein a flight strategy corresponding to thecircling mode comprises: controlling, based on the location information,the UAV to fly circling around the target object based on a specifieddistance.
 15. The imaging control device of claim 13, wherein the flightstrategy corresponding to the spiral mode comprises: controlling, basedon the location information, the UAV to fly circling around the targetobject in a flight path having a shape of a Fibonacci spiral, an equalratio spiral, an equiangular spiral, or an Archimedean spiral, orwherein a flight strategy corresponding to the roaring mode comprises:controlling, based on the location information, the UAV to fly slantlyaccording to a predetermined angle to a first specified locationrelative to the target object, and then controlling the UAV to ascendperpendicular to the ground.
 16. The imaging control device of claim 13,wherein a flight strategy corresponding to the comet circling modecomprises: controlling, based on the location information, the UAV tofly to a second specified location in a direction moving closer to thetarget object, and after circling around the target object starting fromthe second specified location, fly away from the target object.